Fuzzy Matches Research Articles

Can We Standardize Name Reconciliaton via OpenRefine?

Scientific names in biodiversity represent one of the oldest identifiers used in science. As a result, a common repetitive task is being able to reconcile a list of scientific names against curated data sources. Reconciliation allows one to determine if names in a list are spelled correctly, whether they are currently accepted, and their nomenclatural status. There are several online and local resources that provide reconciliation services. We share here the potential in interoperability across reconciliation tools. Global Names Verifier (GNverifier), Catalogue of Life, Global Biodiversity Information Facility (GBIF), Taxonomic Name Resolution Service (TNRS), LifeWatch, National Center for Biotechnology Information (NCBI), World Flora Online, Global Biotic Interactions (GloBI), Nomer, Wikidata, and others provide their own tools for name reconciliation. All these tools have their scope, design decisions, input, and output formats. It is often useful to do reconciliation using several such services, because they often include complementary data. However, with all the idiosyncrasies of services and lack of standardization, it is not an easy task (Islam et al. 2024). It would be great for researchers if all existing and future tools could be standardized. Then moving from one resource to another would be as easy as changing the URL. Implementing elements of Findable, Accessible, Interoperable, and Reusable (FAIR) data management principles would help to create such standards. However, standardizing all existing and future resources to a common interface would be difficult. Some of them have no monetary or programmatic means to modify their code, while others have more urgent priorities. Some resources support a specific research path where adhering to a rigid standard might hinder their innovation. In this paper we suggest interoperability between reconciliation tools by implementing the OpenRefine Reconciliation Service. OpenRefine is a popular and powerful reconciliation and data cleaning application. It is used by many researchers for data transformation and normalization. Any service that implements the OpenRefine Service can be incorporated into data-management workflows just by providing the service's OpenRefine-compatible URL. Such compatible services can easily be discovered by providing their metadata in the OpenRefine Services Registry. In this paper we discuss our implementation of the OpenRefine Service with the Global Names Verifier (GNverifier) reconciliation tool. GNverifier is developed at the Species File Group as a part of the Global Names Architecture initiative. It offers a powerful, configurable, fast way to reconcile scientific names. GNverifier software aggregates data from more than 100 source datasets. Queries return currently accepted names when provided in a dataset. It allows finding matches for names that historically had several suffixes and can do fuzzy and partial matches. It sorts data by many factors to reliably provide the best available results. With a strong focus on software optimization and a sophisticated matching algorithm, it can process 2000 names a second, making it one of the fastest services available. OpenRefine can use GNverifier directly because it is compatible with the OpenRefine protocol. As shown in Fig. 1, switching between GNverifier and Wikidata reconciliation of scientific names requires only change of a service URL. Implementation of the OpenRefine protocol might solve many standardization problems. Some resources already have it implemented (e.g., Wikidata, GNverifier Whitmire and Mozzherin 2023, WFO Plant List, IPNI). Many people already use OpenRefine for their other reconciliation needs. For them, the incorporation of name reconciliation would be especially beneficial because it will fit into their existing data-management workflow Fig. 2. Basic reconciliation by itself is standard by design and a big step forward. Beyond the basic reconciliation (as seen in Fig. 2) there are more data that researchers are interested in. The Service Protocol allows one to add optional "extended" features. For example, for scientific names, we provide "currently accepted" names, data sources where a name was found, taxonomic classification, etc. Fig. 3. To make these fields standardized, we would need a recommendation document that describes additional fields and their format. We need interested parties to participate in its creation and agree on its usage. The same would apply to optional input filters, for example for restricting reconciliation to certain data sources or higher taxonomic entities. We think OpenRefine would be a significant step forward for standardization between name-reconciliation tools.

APCalign: an R package workflow and app for aligning and updating flora names to the Australian Plant Census

Here we present ‘APCalign’, an R package and accompanying browser-sourced application to align and update scientific names for Australian vascular plants to the most likely currently accepted name in the Australian Plant Census (APC) or a name in the Australian Plant Names Index (APNI). Scientific names are the label assigned to unique taxon concepts by the scientific community, but this common terminology is most useful if a taxon concept is consistently referred to by the same name. These links can be broken because of either spelling mistakes or taxonomic changes. Automated tools are required to resolve taxon lists, aligning and updating long lists of possibly erroneous scientific names to the most likely currently accepted names. It is essential that tools specific to the APC/APNI be developed, because these lists specify an endorsed national-level nomenclature used in government legislation and include the uniquely Australian concept of phrase names, absent in global taxonomic datasets. To align input names to names within the APC or APNI, ‘APCalign’ works progressively through a sequence of checks that combine different permutations of the input name, exact versus fuzzy matches, matches that consider the entire name input versus a subset of words, and character strings that indicate a name can be resolved only to a genus or family. The aligned names are then, when possible, updated to a currently accepted taxon concept within the APC. This package should facilitate all research outputs that require diverse scientific name lists to be merged or outdated lists to be updated.

A search tool based on language modelling developed for The Index of Middle English Prose

Non-standardised early vernaculars present a problem for search tools due to the high degree of variation. The challenge lies in the variation found in orthography, syntax, and lexicon between titles, incipits, and explicits in manuscript copies of the same work. Traditional search methods relying on exact string matching or regular expressions fail to address these variations comprehensively. This project presents a web-based search tool specifically designed to handle linguistic and textual variation. The software is made available as a part of the Index of Middle English Prose (IMEP). The search tool addresses the issue of variation by utilizing a database of incipits and explicits, character-based n-gram language models (LMs) built with the Stanford Research Institute Language Modelling (SRILM) toolkit, and a fuzzy search script (IMEP: FSS) written in Python. The tool optimizes for recall, retrieving multiple potential matches for a search string, without attempting to identify the ‘correct’ one. The search process involves looking up exact matches in the database while simultaneously using the fuzzy search script to evaluate the incipits and explicits against a model of the search string, followed by a match of the search string against models of the incipits and explicits. This two-step process shortens the processing time, which would otherwise be unreasonably long, because while using SRILM to match the search string against each incipit or explicit in the IMEP for precision could be time-consuming, running a first step where all texts are matched against a single LM built from the search string allows for faster processing. A web application, built using Django and Docker, combines the results of the direct database lookup and the fuzzy search script, presenting them as a list with exact matches followed by fuzzy matches ordered by increasing model perplexity. The tool is made available Open Access and can be adapted to other datasets.

A search tool based on language modelling developed for The Index of Middle English Prose.

Non-standardised early vernaculars present a problem for search tools due to the high degree of variation. The challenge lies in the variation found in orthography, syntax, and lexicon between titles, incipits, and explicits in manuscript copies of the same work. Traditional search methods relying on exact string matching or regular expressions fail to address these variations comprehensively. This project presents a web-based search tool specifically designed to handle linguistic and textual variation. The software is made available as a part of the Index of Middle English Prose (IMEP). The search tool addresses the issue of variation by utilizing a database of incipits and explicits, character-based n-gram language models (LMs) built with the Stanford Research Institute Language Modelling (SRILM) toolkit, and a fuzzy search script (IMEP: FSS) written in Python. The tool optimizes for recall, retrieving multiple potential matches for a search string, without attempting to identify the 'correct' one. The search process involves looking up exact matches in the database while simultaneously using the fuzzy search script to evaluate the incipits and explicits against a model of the search string, followed by a match of the search string against models of the incipits and explicits. This two-step process shortens the processing time, which would otherwise be unreasonably long, because while using SRILM to match the search string against each incipit or explicit in the IMEP for precision could be time-consuming, running a first step where all texts are matched against a single LM built from the search string allows for faster processing. A web application, built using Django and Docker, combines the results of the direct database lookup and the fuzzy search script, presenting them as a list with exact matches followed by fuzzy matches ordered by increasing model perplexity. The tool is made available Open Access and can be adapted to other datasets.

The Large Taxon Collider: Building the Atlas of Living Australia's taxonomic backbone

One of the most important components of the Atlas of Living Australia (ALA) is our taxonomic backbone, comprised of an index of species names with a suite of web services for matching names and delivering information for our species pages. The index is built by merging lists from multiple Australian authoritative taxonomic sources into a single taxonomic tree. Where the primary data sources are incomplete, we attempt to pad out missing genera and species with alternative sources, for example, using Catalog of Life for some fungi branches, and classification of kingdoms: Viruses, Chromista, Protozoa and Bacteria. When the ALA ingests occurrence records or species checklists, we attempt to match the supplied names against the index using our name matching service. All going well, the service returns the authoritative persistent identifiers for the matched taxon concept and all of the parent taxon concepts. The name matching service will attempt fuzzy and higher rank matches in an attempt to ensure that every record in the ALA finds a home in the taxonomic tree. The merging and matching algorithms are available as open-source code. This presentation will give a brief overview of the construction of our names index, the technical and taxonomic challenges the ALA has faced in building and delivering the service, and our development plans for the future.

Abstract 4092: Fuzzion2: Fast, sensitive detection of known gene fusions by fuzzy pattern matching for clinical testing and large-scale data mining

Abstract Detection of gene fusions is important for discovery of cancer drivers and clinical oncology testing, but existing software tools for fusion detection usually take hours to run and may fail to find lowly expressed fusions. To overcome these limitations, we developed the Fuzzion2 program, which uses pattern matching to detect known gene fusions in unmapped paired-read RNA-Seq data. Given a set of patterns representing fusion transcript breakpoints, Fuzzion2 finds every read pair matching any of the patterns. Both exact and inexact (fuzzy) matches are detected; the fuzzy matching tolerates variations caused by sequencing errors, SNVs, and indels. By employing a novel index of frequency minimizers, Fuzzion2 needs only minutes to process a sample. We have also developed pipelines to produce patterns for Fuzzion2, from fusion contig sequences, from genomic breakpoints in DNA and RNA, and from fusion protein sequences. To evaluate its applicability in clinical testing, we ran Fuzzion2 on ~2,000 RNA-seq samples profiled by the St. Jude clinical genomics program and confirmed its sensitivity in identifying lowly expressed fusions, such as KIAA1549-BRAF in low-grade glioma, which are frequently missed by commonly used fusion detection programs. Notably, Fuzzion2 detected a subclonal BCR-ABL1 fusion expressed at 1% and 6% of the wild-type BCR and ABL1 transcription level, respectively, in a B-lineage ALL sample that also has an IGH-CRLF fusion. Processing RNA-seq data from BCR-ABL1 cell lines, K562 with p210 fusion and OP1 with p190 fusion, diluted at 1:10, 1:100, and 1:1000 showed that Fuzzion2 can detect the fusion at 1:10-1:100 dilution, achieving a sensitivity 10 times greater than that of other fusion detection programs. We also evaluated the performance of Fuzzion2 for large-scale data mining in a study to compare the prevalence of gene fusions in pediatric versus adult cancers. We assembled a set of 15,474 patterns representing 5,480 fusions identified in the Pediatric Cancer Genome Project, NCI TARGET, clinical sequencing, and the COSMIC database. Fuzzion2 was deployed to the NCI Cancer Genomics Cloud and analyzed 9,464 TCGA RNA-seq samples from adult solid and brain tumors. Processing took an average of 6 minutes at a cost of only US$0.16 per sample. Among the 105 recurrent fusions identified in pediatric cancers, only 11 were also found in adult cancers. These shared fusions can be classified into two categories: 1) gene fusions present in cancers that occur in both children and young adults, e.g., synovial sarcoma, papillary thyroid cancer, and fibrolamellar hepatocellular carcinoma; and 2) kinase fusions involving ABL1, NTRK, and FGFR. Our experience with Fuzzion2 demonstrates that it is a powerful tool for time-critical clinical application and large-scale data mining. It is publicly available at https://github.com/stjude/fuzzion2. Citation Format: Stephen V. Rice, Michael N. Edmonson, Liqing Tian, Michael Rusch, David A. Wheeler, Jennifer L. Neary, Scott Newman, Lu Wang, Patrick R. Blackburn, Michael Macias, Andrew Thrasher, Jian Wang, Mark R. Wilkinson, Xin Zhou, Jinghui Zhang. Fuzzion2: Fast, sensitive detection of known gene fusions by fuzzy pattern matching for clinical testing and large-scale data mining [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 4092.

Private identity agreement for private set functionalities1

Private set intersection and related functionalities are among the most prominent real-world applications of secure multiparty computation. While such protocols have attracted significant attention from the research community, other functionalities are often required to support a PSI application in practice. For example, in order for two parties to run a PSI over the unique users contained in their databases, they might first invoke a support functionality to agree on the primary keys to represent their users. This paper studies a secure approach to agreeing on primary keys. We introduce and realize a functionality that computes a common set of identifiers based on incomplete information held by two parties, which we refer to as private identity agreement, and we prove the security of our protocol in the honest-but-curious model. We explain the subtleties in designing such a functionality that arise from privacy requirements when intending to compose securely with PSI protocols. We also argue that the cost of invoking this functionality can be amortized over a large number of PSI sessions, and that for applications that require many repeated PSI executions, this represents an improvement over a PSI protocol that directly uses incomplete or fuzzy matches.

Evaluating the Impact of Integrating Similar Translations into Neural Machine Translation

Previous research has shown that simple methods of augmenting machine translation training data and input sentences with translations of similar sentences (or fuzzy matches), retrieved from a translation memory or bilingual corpus, lead to considerable improvements in translation quality, as assessed by a limited set of automatic evaluation metrics. In this study, we extend this evaluation by calculating a wider range of automated quality metrics that tap into different aspects of translation quality and by performing manual MT error analysis. Moreover, we investigate in more detail how fuzzy matches influence translations and where potential quality improvements could still be made by carrying out a series of quantitative analyses that focus on different characteristics of the retrieved fuzzy matches. The automated evaluation shows that the quality of NFR translations is higher than the NMT baseline in terms of all metrics. However, the manual error analysis did not reveal a difference between the two systems in terms of total number of translation errors; yet, different profiles emerged when considering the types of errors made. Finally, in our analysis of how fuzzy matches influence NFR translations, we identified a number of features that could be used to improve the selection of fuzzy matches for NFR data augmentation.

Towards a Better Integration of Fuzzy Matches in Neural Machine Translation through Data Augmentation

We identify a number of aspects that can boost the performance of Neural Fuzzy Repair (NFR), an easy-to-implement method to integrate translation memory matches and neural machine translation (NMT). We explore various ways of maximising the added value of retrieved matches within the NFR paradigm for eight language combinations, using Transformer NMT systems. In particular, we test the impact of different fuzzy matching techniques, sub-word-level segmentation methods and alignment-based features on overall translation quality. Furthermore, we propose a fuzzy match combination technique that aims to maximise the coverage of source words. This is supplemented with an analysis of how translation quality is affected by input sentence length and fuzzy match score. The results show that applying a combination of the tested modifications leads to a significant increase in estimated translation quality over all baselines for all language combinations.

Fuzzy-Match Repair Guided by Quality Estimation

Computer-aided translation tools based on translation memories are widely used to assist professional translators. A translation memory (TM) consists of a set of translation units (TU) made up of source- and target-language segment pairs. For the translation of a new source segment s', these tools search the TM and retrieve the TUs (s,t) whose source segments are more similar to s'. The translator then chooses a TU and edit the target segment t to turn it into an adequate translation of s'. Fuzzy-match repair (FMR) techniques can be used to automatically modify the parts of t that need to be edited. We describe a language-independent FMR method that first uses machine translation to generate, given s' and (s,t), a set of candidate fuzzy-match repaired segments, and then chooses the best one by estimating their quality. An evaluation on three different language pairs shows that the selected candidate is a good approximation to the best (oracle) candidate produced and is closer to reference translations than machine-translated segments and unrepaired fuzzy matches ( t). In addition, a single quality estimation model trained on a mix of data from all the languages performs well on any of the languages used.

Productivity and quality when editing machine translation and translation memory outputs: an empirical analysis of English to Welsh translation

AbstractThis article reports on a controlled study carried out to examine the possible benefits of editing Machine Translation and Translation Memory outputs when translating from English to Welsh. Using software capable of timing the translation process per segment, 8 professional translators each translated 75 sentences of differing match percentage, and post- edited a further 25 segments of Machine Translation. Basing the final analysis on 800 sentences and 17,440 words, the use of Fuzzy Matches in the 70-99% match range, Exact Matches and Statistical Machine Translation was found to significantly speed up the translation process. Significant correlations were also found between the processing time data of Exact Matches and Machine Translation post-editing, rather than between Fuzzy Matches and Machine Translation as expected. Two experienced translators were then asked to rate all translations for fidelity, grammaticality and style, whereby it was found that the use of translation technology either did not negatively affect translation quality compared to manual translation, or its use actually improved final quality in some cases. As well as confirming the findings of research in relation to translation technology, these findings also contradict supposed similarities between translation quality in terms of style and post-editing Machine Translation.

What does Translation Memory do to translation? The effect of Translation Memory output on specific aspects of the translation process

This article reports on a key-logging experiment carried out in order to investigate the effect that Translation Memory matches in the 70%-95% range have on particular aspects of the translation process. Operationalising the translation process as text (re)production following Englund-Dimitrova (2005), Translog-II is used to investigate whether the use of fuzzy matches in this range can reduce cognitive effort based on Working Memory Capacity and recorded pauses, to study the effect that adapting and correcting fuzzy matches in this range has on linear and non-linear writing processes, and to examine variables related to revision, time and productivity. Results show that initial reading time and self-revision is longer in the case of fuzzy match correction compared to manual translation. Data also show however that cognitive load as measured by pauses is reduced and that productivity is also increased. Significant differences are also observed in terms of text production strategies between the translators who edited the fuzzy matches and those who translated without them.

Correlations between productivity and quality when post-editing in a professional context

This article presents results on the correlation between machine-translated and fuzzy matches segments in terms of productivity and final quality in the context of a localization project. In order to explore these two aspects, we set up an experiment with a group of twenty four professional translators using an online post-editing tool and a customized Moses machine translation engine with a BLEU score of 0.60. The translators were asked to translate from English to Spanish, working on no-match, machine-translated and translation memory segments from the 85---94 % value, using a post-editing tool, without actually knowing if the segment came from machine translation or from translation memory. The texts were corrected by three professional reviewers to assess the final quality of the assignment. The findings suggest that translators have higher productivity and quality when using machine-translated output than when translating without it, and that this productivity and quality is not significantly different from the values obtained when processing fuzzy matches from translation memories in the range 85---94 %.

Genome data classification based on fuzzy matching

Genomic data mining and knowledge extraction is an important problem in bioinformatics. Some research work has been done on unknown genome identification and is based on exact pattern matching of n-grams. In most of the real world biological problems exact matching may not give desired results and the problem in using n-grams is exponential explosion. In this paper we propose a method for genome data classification based on approximate matching. The algorithm works by selecting random samples from the genome database. Tolerance is allowed by generating candidates of varied length to query from these sample sequences. The Levenshtein distance is then checked for each candidate and whether they are k-fuzzily equal. The total number of fuzzy matches for each sequence is then calculated. This is then classified using the data mining techniques namely, naive Bayes, support vector machine, back propagation and also by nearest neighbor. Experiment results are provided for different tolerance levels and they show that accuracy increases as tolerance does. We also show the effect of sampling size on the classification accuracy and it was observed that classification accuracy increases with sampling size. Genome data of two species namely Yeast and E. coli are used to verify proposed method.

Machines, translations and memories: language transfer in the web browser

Classic translation memory (TM) revolutionised technical translation in the 1990s. ‘Enhanced’ TM will revolutionise most types of translation this decade. The massive memory databases collected over 20 years can now be accessed from TM tools for exact and fuzzy matches. Sub-segmental information (terminology and phraseology) can also be gained from them via manual or automated concordancing. Stripped of metadata and formatting information, they have made possible the swift development of statistical machine translation (MT), which can be integrated with TM to provide matches where the memories themselves cannot. At the same time, cloud computing facilitates the adoption of web-based MT (and of web-based TM and translation management systems). Postediting, previously limited to a narrow section of the industry, becomes mainstream. The Read/Write Web demands (and allows for) a new model of translation, with dynamic, user-generated content not well suited to the conventional translation-editing-proofreading model. The web browser seems to drive those who want to make technical translation a profession into either semi-skilled workers enduring quasi sweatshop conditions, or else into the highly-skilled language engineer who helps make the decisions on which tool or process those translators will use for which task.

Shift in Controlled English Norms for Different Purposes and for Different Machine Translation Systems

This research identifies different controlled English (CE) norms to be followed in technical writing for a variety of purposes and for different machine translation (MT) systems. The results of the investigation show that CE norms for MT application are stricter than those for communicative reading. The primary inference here is that human beings can interpret the meanings of polysemous words, pronouns, prepositional phrases based on the context and easily detect the misspellings, but MT systems fail to do so. In addition, a comparison of CE norms for the application of two MT systems indicates that the corpus-based Google MT is less constrained than rule-based TransWhiz in the lexical area. This phenomenon is attributable to the selection of a highly probabilistic module as the semantic scoring preference for the suggested translation provided by Google MT, not word-for-word translation by TransWhiz. In contrast, Google MT is more constrained than TransWhiz in the syntactic area. The inference is that TransWhiz parses syntactic constructions and transfers the parsing result based on grammatical rules stored in the MT system, so it may modify the original word sequence to make the translation conform to linguistic patterns in the target language. Contrary to this, Google MT depends on fuzzy or exact matches statistically retrieved from the labeled corpus. If no matches can be found, syntactically inappropriate translations will be produced. Seen in this regard, CE norms are never fixed and have to be modified through the evolution of time and MT technology.

Time-of-use Price Optimizing Model And Fuzzy Solving Method

Abstract Establishing appropriate Time-Of-Use (TOU) power price can shift peak load and optimize resources in society. The historical data of prices and consumptive quantities of TOU price measures are analyzed, and the reaction curve of consumers are constructed. On the basis of this, optimization model of TOU price is established on the demand side. Because optimization of min (max) and multi-objects are present in the model, to solve it, firstly transforming the objects into the restriction condition, which will construct the new multi-objects non-linear model, and then, the most satisfactory method in the fuzzy matches is adopted to solve the multi-objects model. At last, the example shows that the method is feasible and it supplies references to the pricing mechanism of TOU power price, which is beneficial to push this measure.

ANSWER: APPROXIMATE NAME SEARCH WITH ERRORS IN LARGE DATABASES BY A NOVEL APPROACH BASED ON PREFIX-DICTIONARY

The obvious need for using modern computer networking capabilities to enable the effective sharing of information has resulted in data-sharing systems, which store, and manage large amounts of data. These data need to be effectively searched and analyzed. More specifically, in the presence of dirty data, a search for specific information by a standard query (e.g., search for a name that is misspelled or mistyped) does not return all needed information, as required in homeland security, criminology, and medical applications, amongst others. Different techniques, such as soundex, phonix, n-grams, edit-distance, have been used to improve the matching rate in these name-matching applications. These techniques have demonstrated varying levels of success, but there is a pressing need for name matching approaches that provide high levels of accuracy in matching names, while at the same time maintaining low computational complexity. In this paper, such a technique, called ANSWER, is proposed and its characteristics are discussed. Our results demonstrate that ANSWER possesses high accuracy, as well as high speed and is superior to other techniques of retrieving fuzzy name matches in large databases.

Teaching non-hierarchical classification

The importance of classificatory taxonomy and its role in teaching is discussed. A computeraided training tool, called CLASSIFIER, for teaching non-hierarchical classification is described which provides both exact and fuzzy matches with neighbouring species types. The tool is general and may be applied to different domains. The success of this tool for teaching the taxonomy of Glossopteris plant fossils and of gems is analysed.