Spelling Correction System Research Articles

Improving the quality of Persian clinical text with a novel spelling correction system

BackgroundThe accuracy of spelling in Electronic Health Records (EHRs) is a critical factor for efficient clinical care, research, and ensuring patient safety. The Persian language, with its abundant vocabulary and complex characteristics, poses unique challenges for real-word error correction. This research aimed to develop an innovative approach for detecting and correcting spelling errors in Persian clinical text.MethodsOur strategy employs a state-of-the-art pre-trained model that has been meticulously fine-tuned specifically for the task of spelling correction in the Persian clinical domain. This model is complemented by an innovative orthographic similarity matching algorithm, PERTO, which uses visual similarity of characters for ranking correction candidates.ResultsThe evaluation of our approach demonstrated its robustness and precision in detecting and rectifying word errors in Persian clinical text. In terms of non-word error correction, our model achieved an F1-Score of 90.0% when the PERTO algorithm was employed. For real-word error detection, our model demonstrated its highest performance, achieving an F1-Score of 90.6%. Furthermore, the model reached its highest F1-Score of 91.5% for real-word error correction when the PERTO algorithm was employed.ConclusionsDespite certain limitations, our method represents a substantial advancement in the field of spelling error detection and correction for Persian clinical text. By effectively addressing the unique challenges posed by the Persian language, our approach paves the way for more accurate and efficient clinical documentation, contributing to improved patient care and safety. Future research could explore its use in other areas of the Persian medical domain, enhancing its impact and utility.

PERCORE: A Deep Learning-Based Framework for Persian Spelling Correction with Phonetic Analysis

This research introduces a state-of-the-art Persian spelling correction system that seamlessly integrates deep learning techniques with phonetic analysis, significantly enhancing the accuracy and efficiency of natural language processing (NLP) for Persian. Utilizing a fine-tuned language representation model, our methodology effectively combines deep contextual analysis with phonetic insights, adeptly correcting both non-word and real-word spelling errors. This strategy proves particularly effective in tackling the unique complexities of Persian spelling, including its elaborate morphology and the challenge of homophony. A thorough evaluation on a wide-ranging dataset confirms our system’s superior performance compared to existing methods, with impressive F1-Scores of 0.890 for detecting real-word errors and 0.905 for correcting them. Additionally, the system demonstrates a strong capability in non-word error correction, achieving an F1-Score of 0.891. These results illustrate the significant benefits of incorporating phonetic insights into deep learning models for spelling correction. Our contributions not only advance Persian language processing by providing a versatile solution for a variety of NLP applications but also pave the way for future research in the field, emphasizing the critical role of phonetic analysis in developing effective spelling correction system.

How to Progressively Build Thai Spelling Correction Systems?

How to Progressively Build Thai Spelling Correction Systems?

Amazigh spell checker using Damerau-Levenshtein algorithm and N-gram

Natural language processing (NLP) is a rapidly growing research field in computer science and cognitive science. Automatic correction of lexical errors is one of the applications of NLP, which aims to detect errors in the text and suggest possible corrections based on computer and linguistic models. After a state of the art and a comparison of spelling correction approaches, we developed a spelling error correction system for the Amazigh language combining the Damerau-Levenshtein algorithm and N-gram. This tool will propose possible corrections for each misspelled word in the text. Successful tests have been carried out using an Amazigh corpus.

Learning from mistakes: Improving spelling correction performance with automatic generation of realistic misspellings

AbstractSequence to sequence models (seq2seq) require a large amount of labelled training data to learn the mapping between the input and output. A large set of misspelled words together with their corrections is needed to train a seq2seq spelling correction system. Low‐resource languages such as Turkish usually lack such large annotated datasets. Although misspelling‐reference pairs can be synthesized with a random procedure, the generated dataset may not well match to genuine human‐made misspellings. This might degrade the performance in realistic test scenarios. In this paper, we propose a novel procedure to automatically introduce human‐like misspellings to legitimate words in Turkish language. Generated human‐like misspellings are used to improve the performance of a seq2seq spelling correction system. The proposed system consists of two separate models; a misspelling generator and a spelling corrector. The generator is trained using a relatively small number of human‐made misspellings and their manual corrections. Reference words and their misspellings are used as inputs and outputs of the generator, respectively. As a result, it is trained to add realistic spelling errors to the valid words. Training data of the spelling corrector is augmented by the generator's human‐like misspellings. In the experiments, we observe that the data augmentation significantly improves the spelling correction performance. Our proposed method yields 5% absolute improvement over the state‐of‐the‐art Turkish spelling correction systems in a test set which contains human‐made misspellings from Twitter messages.

Survey of Automatic Spelling Correction

Automatic spelling correction has been receiving sustained research attention. Although each article contains a brief introduction to the topic, there is a lack of work that would summarize the theoretical framework and provide an overview of the approaches developed so far. Our survey selected papers about spelling correction indexed in Scopus and Web of Science from 1991 to 2019. The first group uses a set of rules designed in advance. The second group uses an additional model of context. The third group of automatic spelling correction systems in the survey can adapt its model to the given problem. The summary tables show the application area, language, string metrics, and context model for each system. The survey describes selected approaches in a common theoretical framework based on Shannon’s noisy channel. A separate section describes evaluation methods and benchmarks.

Context-Dependent Sequence-to-Sequence Turkish Spelling Correction

In this article, we make use of sequence-to-sequence (seq2seq) models for spelling correction in the agglutinative Turkish language. In the baseline system, misspelled and target words are split into their letters and the letter sequences are fed into the seq2seq model. We prefer letters as the unit of the model due to the agglutinative nature of Turkish, which results in an impractical dictionary size when words are used as a dictionary unit. In order to improve the baseline performance, we incorporate right and left context of the misspelled words. All context words are represented with their first three consonants in the context-dependent model. We train the seq2seq models using a large text corpus collected automatically from the Internet. The corpus contains approximately 4 million sentences. We randomly introduce substitution, deletion, and insertion spelling errors to the words in the corpus. We test the performance of the proposed context-dependent seq2seq model using synthetic and realistic test sets. The synthetic test set is constructed similar to the training set. The realistic test set contains human-made misspellings from Twitter messages. In the experiments, we observed that the proposed context-dependent model performs significantly better than the baseline system. Its correction accuracy reaches 94% on the synthetic dataset. Additionally, the proposed method provides 2.1% absolute improvement over a state-of-the-art Turkish spelling correction system on the Twitter test set.

Design and implementation of Persian spelling detection and correction system based on Semantic

Design and implementation of Persian spelling detection and correction system based on Semantic

Fast and Accurate Spelling Correction Using Trie and Damerau-levenshtein Distance Bigram

This research was intended to create a fast and accurate spelling correction system with the ability to handle both kind of spelling errors, non-word and real word errors. Existing spelling correction system was analyzed and was then applied some modifications to improve its accuracy and speed. The proposed spelling correction system is then built based on the method and intuition used by existing system along with the modifications made in previous step. The result is a various spelling correction system using different methods. Best result is achieved by the system that uses bigram with Trie and Damerau-Levenshtein distance with the word level accuracy of 84.62% and an average processing speed of 18.89 ms per sentence.

Learning string distance with smoothing for OCR spelling correction

Large databases of scanned documents (medical records, legal texts, historical documents) require natural language processing for retrieval and structured information extraction. Errors caused by the optical character recognition (OCR) system increase ambiguity of recognized text and decrease performance of natural language processing. The paper proposes OCR post correction system with parametrized string distance metric. The correction system learns specific error patterns from incorrect words and common sequences of correct words. A smoothing technique is proposed to assign non-zero probability to edit operations not present in the training corpus. Spelling correction accuracy is measured on database of OCR legal documents in English language. Language model and learning string metric with smoothing improves Viterbi-based search for the best sequence of corrections and increases performance of the spelling correction system.

Perspell: A New Persian Semantic-Based Spelling Correction System

Due to the special features of Persian, developing natural language processing tools for it involves an array of challenges. Lack of efficient Persian knowledge sources is another obstacle to research this language. The goal of this article was to overcome these problems by implementing spelling correction task. The main outputs of this study included a parallel corpus, an N-gram language model for Persian, and a semantic-based spelling correction system named Perspell, which made use of extracted language model. Compared to its rival software (including Vafa spellchecker), Perspell could detect and correct nonword and real word errors more successfully. The rate of real word error detection in Perspell was 95%. In fact, its outstanding ability to detect real word errors as well as its significant improvement in terms of F-measure were the two advantages of the proposed system.

Context-Sensitive Spelling Correction of Consumer-Generated Content on Health Care.

BackgroundConsumer-generated content, such as postings on social media websites, can serve as an ideal source of information for studying health care from a consumer’s perspective. However, consumer-generated content on health care topics often contains spelling errors, which, if not corrected, will be obstacles for downstream computer-based text analysis.ObjectiveIn this study, we proposed a framework with a spelling correction system designed for consumer-generated content and a novel ontology-based evaluation system which was used to efficiently assess the correction quality. Additionally, we emphasized the importance of context sensitivity in the correction process, and demonstrated why correction methods designed for electronic medical records (EMRs) failed to perform well with consumer-generated content.MethodsFirst, we developed our spelling correction system based on Google Spell Checker. The system processed postings acquired from MedHelp, a biomedical bulletin board system (BBS), and saved misspelled words (eg, sertaline) and corresponding corrected words (eg, sertraline) into two separate sets. Second, to reduce the number of words needing manual examination in the evaluation process, we respectively matched the words in the two sets with terms in two biomedical ontologies: RxNorm and Systematized Nomenclature of Medicine -- Clinical Terms (SNOMED CT). The ratio of words which could be matched and appropriately corrected was used to evaluate the correction system’s overall performance. Third, we categorized the misspelled words according to the types of spelling errors. Finally, we calculated the ratio of abbreviations in the postings, which remarkably differed between EMRs and consumer-generated content and could largely influence the overall performance of spelling checkers.ResultsAn uncorrected word and the corresponding corrected word was called a spelling pair, and the two words in the spelling pair were its members. In our study, there were 271 spelling pairs detected, among which 58 (21.4%) pairs had one or two members matched in the selected ontologies. The ratio of appropriate correction in the 271 overall spelling errors was 85.2% (231/271). The ratio of that in the 58 spelling pairs was 86% (50/58), close to the overall ratio. We also found that linguistic errors took up 31.4% (85/271) of all errors detected, and only 0.98% (210/21,358) of words in the postings were abbreviations, which was much lower than the ratio in the EMRs (33.6%).ConclusionsWe conclude that our system can accurately correct spelling errors in consumer-generated content. Context sensitivity is indispensable in the correction process. Additionally, it can be confirmed that consumer-generated content differs from EMRs in that consumers seldom use abbreviations. Also, the evaluation method, taking advantage of biomedical ontology, can effectively estimate the accuracy of the correction system and reduce manual examination time.

Automated misspelling detection and correction in clinical free-text records

Accurate electronic health records are important for clinical care and research as well as ensuring patient safety. It is crucial for misspelled words to be corrected in order to ensure that medical records are interpreted correctly. This paper describes the development of a spelling correction system for medical text. Our spell checker is based on Shannon’s noisy channel model, and uses an extensive dictionary compiled from many sources. We also use named entity recognition, so that names are not wrongly corrected as misspellings. We apply our spell checker to three different types of free-text data: clinical notes, allergy entries, and medication orders; and evaluate its performance on both misspelling detection and correction. Our spell checker achieves detection performance of up to 94.4% and correction accuracy of up to 88.2%. We show that high-performance spelling correction is possible on a variety of clinical documents.

Spelling Error Patterns in Brazilian Portuguese

Fifty years after Damerau set up his statistics for the distribution of errors in typed texts, his findings are still used in a range of different languages. Because these statistics were derived from texts in English, the question of whether they actually apply to other languages has been raised. We address this issue through the analysis of a set of typed texts in Brazilian Portuguese, deriving statistics tailored to this language. Results show that diacritical marks play a major role, as indicated by the frequency of mistakes involving them, thereby rendering Damerau's original findings mostly unfit for spelling correction systems, although still holding them useful, should one set aside such marks. Furthermore, a comparison between these results and those published for Spanish show no statistically significant differences between both languages—an indication that the distribution of spelling errors depends on the adopted character set rather than the language itself.

A Correction Model for Real-word Errors

Spell Checker is used to identify and correct mistakes made by users while writing text and the mistakes are generally spelling mistakes. An intelligent spelling correction system SMC is proposed to automatically correct spelling mistakes in text-editor or text documents using contextual information of the confused words. The system is capable to correct words belonging to the set of confused words fed into it if they are contextually wrong. In this technique, an algorithm to identify and correct real-word errors is proposed. One phase of algorithm uses trigram approach to correct spelling mistakes and the other phase of algorithm uses Bayesian approach to correct spelling mistakes. Brown corpus is used as a training set and a set of commonly confused words is used in this case. Selection of words in other phase of algorithm uses synonyms derived from dictionary in the scenario when words are not found in the corpus. Comparative analysis of the proposed approach with tribayes has also been performed to identify the accuracy of SMC. The results indicate that SMC gives higher accuracy for spelling mistakes identification and correction for the commonly confused words as compared to other spelling correction algorithms.

FarsiSpell: A spell-checking system for Persian using a large monolingual corpus

In recent years, great availability of various language resources in different forms as well as rapid development of computer technology and programming skills have made researchers in the fields of linguistics and computer science cooperate in solving different problems of computational linguistics and natural language processing. Building large monolingual as well as bilingual corpora in digital forms and storing them in computer memories has enabled linguists and lan- guage engineers to automatically explore techniques for processing information with the help of various computer programs without any need to manually col- lect and analyze data. One of the main applications of monolingual corpora can be seen in developing automatic spell-checking systems. In such systems, a large monolingual corpus can function as a database instead of a monolingual dictionary. In the present study, it has been tried to demonstrate the effectiveness of a large monolingual corpus of Persian in improving the output quality of a spell-checker developed for this language. In the present spelling correction system, the three phases of error detection, making suggestions, and ranking suggestions are performed in three separate stages. An experiment was carried out to evaluate the performance of the spell-checking system.

Character confusion versus focus word-based correction of spelling and OCR variants in corpora

We present a new approach based on anagram hashing to handle globally the lexical variation in large and noisy text collections. Lexical variation addressed by spelling correction systems is primarily typographical variation. This is typically handled in a local fashion: given one particular text string some system of retrieving near-neighbors is applied, where near-neighbors are other text strings that differ from the particular string by a given number of characters. The difference in characters between the original string and one of its retrieved near-neighbors constitutes a particular character confusion. We present a global way of performing this action: for all possible particular character confusions given a particular edit distance, we sequentially identify all the pairs of text strings in the text collection that display a particular confusion. We work on large digitized corpora, which contain lexical variation due to both the OCR process and typographical or typesetting error and show that all these types of variation can be handled equally well in the framework we present. The character confusion-based prototype of Text-Induced Corpus Clean-up (ticcl) is compared to its focus word-based counterpart and evaluated on 6 years’ worth of digitized Dutch Parliamentary documents. The character confusion approach is shown to gain an order of magnitude in speed on its word-based counterpart on large corpora. Insights gained about the useful contribution of global corpus variation statistics are shown to also benefit the more traditional word-based approach to spelling correction. Final tests on a held-out set comprising the 1918 edition of the Dutch daily newspaper ‘Het Volk’ show that the system is not sensitive to domain variation.

Evaluating and reducing the effect of data corruption when applying bag of words approaches to medical records

Unlike journal corpora, which are supposed to be carefully reviewed before being published, the quality of documents in a patient record are often corrupted by misspelled words and conventional graphies or abbreviations. After a survey of the domain, the paper focuses on evaluating the effect of such corruption on an information retrieval (IR) engine. The IR system uses a classical bag of words approach, with stems as representation items and term frequency–inverse document frequency (tf–idf) as weighting schema; we pay special attention to the normalization factor. First results shows that even low corruption levels (3%) do affect retrieval effectiveness (4–7%), whereas higher corruption levels can affect retrieval effectiveness by 25%. Then, we show that the use of an improved automatic spelling correction system, applied on the corrupted collection, can almost restore the retrieval effectiveness of the engine.

OCRSpell: an interactive spelling correction system for OCR errors in text

In this paper, we describe a spelling correction system designed specifically for OCR-generated text that selects candidate words through the use of information gathered from multiple knowledge sources. This system for text correction is based on static and dynamic device mappings, approximate string matching, and n-gram analysis. Our statistically based, Bayesian system incorporates a learning feature that collects confusion information at the collection and document levels. An evaluation of the new system is presented as well.

Natural language interface for an expert system

This paper provides complete analysis of natural language interface from the screen manager to the parser/understander, even though it has some limitations. The main focus is on the design and development of a subsystem for understanding natural language input in an expert system. Fast response time and user‐friendliness were the most important considerations in the design. The screen manager provides an easy editing capability for the user. The spelling correction system can detect most spelling errors and correct them automatically, quickly and effectively. The Lexical Functional Grammar (LFG) parser and the understander are designed to handle most types of simple sentences, fragments, and ellipses. As a result, this system as applied to a man‐machine dialogue gives improved ill‐formed natural language input handling capabilities to the expert system.