Word-level Errors Research Articles

Enhancing Machine Translation Quality Estimation via Fine-Grained Error Analysis and Large Language Model

Fine-grained error span detection is a sub-task within quality estimation that aims to identify and assess the spans and severity of errors present in translated sentences. In prior quality estimation, the focus has predominantly been on evaluating translations at the sentence and word levels. However, such an approach fails to recognize the severity of specific segments within translated sentences. To the best of our knowledge, this is the first study that concentrates on enhancing models for this fine-grained error span detection task in machine translation. This study introduces a framework that sequentially performs sentence-level error detection, word-level error span extraction, and severity assessment. We present a detailed analysis for each of the methodologies we propose, substantiating the effectiveness of our system, focusing on two language pairs: English-to-German and Chinese-to-English. Our results suggest that task granularity enhances performance and that a prompt-based fine-tuning approach can offer optimal performance in the classification tasks. Furthermore, we demonstrate that employing a large language model to edit the fine-tuned model’s output constitutes a top strategy for achieving robust quality estimation performance.

From 2D to 3D

Mid-air text entry on virtual keyboards suffers from the lack of tactile feedback, which brings challenges to both tap detection and input prediction. In this paper, we explored the feasibility of single-finger typing on virtual QWERTY keyboards in mid-air. We first conducted a study to examine users' 3D typing behavior on different sizes of virtual keyboards. Results showed that the participants perceived the vertical projection of the lowest point on the keyboard during a tap as the target location and inferring taps based on the intersection between the finger and the keyboard was not applicable. Aiming at this challenge, we derived a novel input prediction algorithm that took the uncertainty in tap detection into a calculation as probability, and performed probabilistic decoding that could tolerate false detection. We analyzed the performance of the algorithm through a full-factorial simulation. Results showed that the SVM-based probabilistic touch detection together with a 2D elastic probabilistic decoding algorithm (elasticity = 2) could achieve the optimal top-5 accuracy of 94.2%. In the evaluation user study, the participants reached a single-finger typing speed of 26.1 WPM with 3.2% uncorrected word-level error rate, which was significantly better than both tap-based and gesture-based baseline techniques. Also, the proposed technique received the highest preference score from the users, proving its usability in real text entry tasks.

Chinese Grammatical Error Correction Using Pre-trained Models and Pseudo Data

In recent studies, pre-trained models and pseudo data have been key factors in improving the performance of the English grammatical error correction (GEC) task. However, few studies have examined the role of pre-trained models and pseudo data in the Chinese GEC task. Therefore, we develop Chinese GEC models based on three pre-trained models: Chinese BERT, Chinese T5, and Chinese BART, and then incorporate these models with pseudo data to determine the best configuration for the Chinese GEC task. On the natural language processing and Chinese computing (NLPCC) 2018 GEC shared task test set, all our single models outperform the ensemble models developed by the top team of the shared task. Chinese BART achieves an F score of 37.15, which is a state-of-the-art result. We then combine our Chinese GEC models with three kinds of pseudo data: Lang-8 (MaskGEC), Wiki (MaskGEC), and Wiki (Backtranslation). We find that most models can benefit from pseudo data, and BART+Lang-8 (MaskGEC) is the ideal setting in terms of accuracy and training efficiency. The experimental results demonstrate the effectiveness of the pre-trained models and pseudo data on the Chinese GEC task and provide an easily reproducible and adaptable baseline for future works. Finally, we annotate the error types of the development data; the results show that word-level errors dominate all error types, and word selection errors must be addressed even when using pre-trained models and pseudo data. Our codes are available at https://github.com/wang136906578/BERT-encoder-ChineseGEC .

Kurdish Kurmanji Lemmatization and Spell-checker with Spell-correction

There are many studies about using lemmatization and spell-checker with spell-correction regarding English, Arabic, and Persian languages but only few studies found regarding low-resource languages such as Kurdish language and more specifically for Kurmanji dialect, which increased the need of creating such systems. Lemmatization is the process of determining a base or dictionary form (lemma) for a specific surface pattern, whereas spell-checkers and spell-correctors determine whether a word is correctly spelled also correct a range of spelling errors, respectively. This research aims to present a lemmatization and a word-level error correction system for Kurdish Kurmanji Dialect, which are the first tools for this dialect based on our knowledge. The proposed approach for lemmatization is built on morphological rules, and a hybrid approach that relies on the n-gram language model and the Jaccard Coefficient Similarity algorithm was applied to the spell-checker and spell-correction. The process results for lemmatization, as detailed in this article, rates of 97.7% and 99.3% accuracy for noun and verb lemmatization, correspondingly. Furthermore, for spell-checker and spell-correction, accordingly, accuracy rates of 100% and 90.77% are attained.

ENGLISH FIRST ADDITIONAL LANGUAGE WRITING ERRORS OF ISIZULU-SPEAKING LEARNERS IN FET WRITING CLASSES IN KWAZULU-NATAL

Error analysis presupposes a distinction between two languages. This distinction was primarily concerned with how the two languages facilitated or hindered learning of the target language. In its original premise, the juxtaposed languages were considered facilitative if they were similar but restrictive if they were different. This gave rise to the notion of transference where it was viewed either negatively when it hindered acquisition of certain structures or positively if it facilitated acquisition. This approach developed into a new focus of studying errors and analysing them. At first such an analysis focused on mother tongue interference but later developed into interlanguage where errors were viewed as a natural phenomenon that demonstrates the progress of learning. This paper is therefore grounded on error analysis where errors in writing of high school learners are analysed in order to interrogate where second language learners are in internalisation of English structures. The analysis covers grammatical aspects such as morphology, syntax and semantics. The findings indicate that learners’ errors range from word and phrase level to sentence level. At word level errors of morphological constructions and spelling were committed and at phrase level, errors were caused by the complexity of the noun and verb phrases. At sentence level, there were errors in sentence construction. The study has implications for the teaching of writing.

Grammatical Errors in Writing Recount Texts (A Case Study at The First Year of Senior High School)

Writting in a foreign language is one of the most difficult abilities in learning language skills. The problem occurred because understanding grammatical knowledge and rewording sentence writing was lowering frequency by students. This research aimed to investigate grammatical errors made by students and analyze the causes of language interface errors in writing recount text. Thirty respondents were chosen purposively to be involved in this study. The data collection instruments were conducted by writing text and conducting interviews under causes of grammatical errors. Results from the study revealed that 11 distinct grammatical errors were present in the students' writing: word and sentence level errors. The most prevalent grammatical errors at the word level are tense verb errors 69 (35.38%), and punctuation has 39 (20%) problems. Further, 18 (9.23%) word choice problems followed that. In the meantime, the causes of errors were determined to be intralingual (0.22%) and interlingual (0.7%) errors. It implemented that the teacher should be offered an explanation and correction of any grammatical errors made by students, as well as comments regarding whether or not students were successful in learning English writing material and need more practice.

An Error Analysis of Nonword Repetition Performance in Toddlers and Young Children: Theoretical Implications and Clinical Utility.

Purpose Nonword repetition has been cited as a measure of phonological working memory and continues to gain status as a clinical tool used to identify language impairment in school-age children. Less is known about nonword repetition skills in the toddler population. Method The current study presents a detailed analysis of errors by segmenting nonwords into word, syllable, and phoneme levels. Errors were also analyzed for type (e.g., addition, substitution, deletion). The Test of Early Nonword Repetition was used to measure performance in a sample of 36 typically developing children, aged 24-48 months. Clinical assessments including parent report, language sampling procedures, and standardized assessments were also administered. Results As a group, participants produced significantly more syllable errors compared to word-level errors; however, most errors were made at the phoneme level. Errors of addition were the least common error type, and no differences between substitutions or deletions were present for the entire sample. Toddlers (aged 2 years) produced more syllable-level errors compared to older children (aged 3 years). Substitution errors were positively correlated with performance on clinical measures of language, whereas deletion errors were negatively correlated with performance. Conclusion Nonword repetition performance patterns in young children may be associated with language delay or language impairment and have both clinical and theoretical relevance.

Written Language of Children with Specific Language Impairment

Writing is an important academically and socially related language skill. However, few researchers have analyzed the written language samples of children with Specific Language Impairment (SLI). It is unknown if children with SLI produce different types of written language errors than children with typical language (TL). The purpose of this study was to increase understanding of the writing patterns of children with Specific Language Impairment. Specifically, this study analyzed the morphosyntactic and syntactic errors that children with SLI make and if they differ from the errors made by children with TL. Writing samples of children ages 7 to 10 were collected. The SLI group was determined by norm-referenced language assessments such as the Clinical Evaluation of Language Fundamentals – Fourth Edition. These samples were coded for errors based on a coding manual created by the authors of this study. Results revealed that children with SLI make more overall errors in complex sentences than children with TL. The errors made by children with SLI were syntactic word level errors, such as the addition or omission of one word. However, there was not a significant difference between the morphological errors made in writing samples of children with SLI and children with TL. The large age range of children within our SLI group may have impacted our conclusion that morphosyntax may not be a critical marker of SLI in children.

Initial validation of a measure of decoding difficulty as a unique predictor of miscues and passage reading fluency.

Quantifying the decoding difficulty (i.e., 'decodability') of text is important for accurately matching young readers to appropriate text and scaffolding reading development. Since no easily accessible, quantitative, word-level metric of decodability exists, we developed a decoding measure (DM) that can be calculated via a web-based scoring application that takes into account sub-lexical components (e.g. orthographic complexity), thus measuring decodability at the grapheme-phoneme level, which can be used to judge decodability of individual words or passages. Here we report three experiments using the DM: two predicting children's word-level errors and one predicting passage reading fluency. Generalized linear mixed effect models showed that metrics from the DM explained unique variance in children's oral reading miscues after controlling for word frequency in two samples of children (experiments 1 and 2), and that more errors were made on words with higher DM scores for poor readers. Furthermore, the DM metrics predicted children's number of words read correctly per minute after accounting for estimated Lexile passage scores in a third sample (experiment 3). These results show that after controlling for word frequency (experiments 1 and 2) and estimated Lexile scores (experiment 3) the model including the DM metrics was significantly better in predicting children's word reading fluency both for individual words and passages. While further refinement of this DM measure is ongoing, it appears to be a promising new measure of decodability at both the word and passage level. The measure also provides the opportunity to enable precision teaching techniques, as grapheme-phoneme correspondence profiles unique to each child could facilitate individualized instruction, and text.

A Generic Block-Level Error Confinement Technique for Memory Based on Principal Component Analysis

Nanoscale CMOS technology has encountered severe reliability issues especially in on-chip memory. Conventional word-level error resilience techniques such as Error Correcting Codes (ECC) suffer from high physical overhead and inability to correct increasingly reported multiple bit flip errors. On the other hands, state-of-the-art applications such as image processing and machine learning loosen the requirement on the levels of data protection, which result in dedicated techniques of approximated fault tolerance. In this work, we introduce a novel error protection scheme for memory, based on feature extraction through Principal Component Analysis and the modular-wise technique to segment the data before PCA. The extracted features can be protected by replacing the fault vector with the averaged confinement vectors. This approach confines the errors with either single or multi-bit flips for generic data blocks, whilst achieving significant savings on execution time and memory usage compared to traditional ECC techniques. Experimental results of image processing demonstrate that the proposed technique results in a reconstructed image with PSNR over 30 dB, while robust against both single bit and multiple bit flip errors, with reduced memory storage to just 22.4% compared to the conventional ECC-based technique.

Morphosyntactic Development and Severe Parental Neglect in 4-Year-Old French-Speaking Children: ELLAN study.

Language is the most frequently compromised area of development in English-speaking neglected children, particularly the morphosyntactic component of language. This is very worrisome given its central role in academic success and social participation. No previous study has examined the morphosyntactic skills of French-speaking neglected children, despite the morphological richness of French. This study aimed to fill this gap. Forty-four neglected (mean age = 48.32 months, SD = 0.45) and 92 non-neglected (mean age = 48.07 months, SD = 0.24) French-speaking children participated. Measures of morphosyntactic skills were derived from a sample of spontaneous language collected during standardized semistructured play and analyzed using Systematic Analysis of Language Transcripts software (2012) . Four morphosyntactic indicators were compared using analyses of variance and Kolmogorov-Smirnov tests: the mean length of utterances (MLU), verbal inflections, word-level errors, and omission errors. The results indicate that 25.6% of the neglected children presented clinically significant morphosyntactic difficulties, as evidenced by a significantly shorter MLU (M = 5.60, SD = 1.13; M = 6.90, SD = 1.30), fewer verbal inflections, and more frequent word omission errors compared to their non-neglected peers. The results confirm that French-speaking neglected children present many morphosyntactic difficulties. This study argues for sustained speech-language services for these children.

Does a dual-task selectively inhibit the metacognitive activities in text revision?

The effect of cognitive overload on production deficiency in metacognitive activities was investigated using a dual-task paradigm. Undergraduate students (N=54) were assigned to one of two conditions, the dual-task condition and the non-dual-task condition. They completed a text revision task that included two types of problems, word level errors and disorganized construction. The construction improvement was assumed to require metacognitive activities from the readers’ point of view. Participants in the dual-task condition counted the number of auditory presentations of a target word while revising the given text. The results indicated that the performance level of construction improvement was lower in the dual-task condition than in the non-dual-task condition, while there was no difference in word error correction between the two conditions. It is interpreted that metacognitive activities, such as monitoring and control in text revision for comprehensibility from the readers’ perspective, are selectively impaired by cognitive overload. This study experimentally demonstrated that cognitive overload caused a production deficiency in metacognitive activities. Methodological issues are discussed for future research.

Measuring Machine Translation Errors in New Domains

We develop two techniques for analyzing the effect of porting a machine translation system to a new domain. One is a macro-level analysis that measures how domain shift affects corpus-level evaluation; the second is a micro-level analysis for word-level errors. We apply these methods to understand what happens when a Parliament-trained phrase-based machine translation system is applied in four very different domains: news, medical texts, scientific articles and movie subtitles. We present quantitative and qualitative experiments that highlight opportunities for future research in domain adaptation for machine translation.

Hjerson: An Open Source Tool for Automatic Error Classification of Machine Translation Output

We describe Hjerson, a tool for automatic classification of errors in machine translation output. The tool features the detection of five word level error classes: morphological errors, reodering errors, missing words, extra words and lexical errors. As input, the tool requires original full form reference translation(s) and hypothesis along with their corresponding base forms. It is also possible to use additional information on the word level (e.g.  tags) in order to obtain more details. The tool provides the raw count and the normalised score (error rate) for each error class at the document level and at the sentence level, as well as original reference and hypothesis words labelled with the corresponding error class in text and  formats. 1. Motivation Human error classification and analysis of machine translation output presented in (Vilar et al., 2006) have become widely used in recent years in order to get detailed answers about strengths and weaknesses of a translation system. Another types of human error analysis have also been carried out, e.g. (Farrus et al., 2009) suitable for the Spanish and Catalan languages. However, human error classification is a difficult and time consuming task, and automatic methods are needed. Hjerson is a tool for automatic error classification which systematically covers the main word level error categories defined in (Vilar et al., 2006): morphological (inflectional) errors, reordering errors, missing words, extra words and lexical errors. It implements the method based on the standard word error rate () combined with the precision and recall based error rates (Popovic and Ney, 2007) and it has been © 2011 PBML. All rights reserved. Corresponding author: maja.popovic@dfki.de Cite as: Maja Popovic. Hjerson: An Open Source Tool for Automatic Error Classification of Machine Translation Output. The Prague Bulletin of Mathematical Linguistics No. 96, 2011, pp. 59–67. doi: 10.2478/v10108-011-0011-4. PBML 96 OCTOBER 2011 tested on various language pairs and tasks. It is shown that the obtained results have high correlation (between 0.6 and 1.0) with the results obtained by human evaluators (Popovic and Burchardt, 2011; Popovic and Ney, 2011). The tool is written in Python, and is available under an open-source licence. We hope that the release of the toolkit will facilitate the error analysis and classification for the researchers, and also stimulate further development of the proposed method.

Design of language models at various phases of Tamil speech recognition system

This paper describes the use of language models in various phases of Tamil speech recognition system for improving its performance. In this work, the language models are applied at various levels of speech recognition such as segmentation phase, recognition phase and the syllable and word level error correction phase. The speech signals were segmented at phonetic level based on their acoustic characteristics. The wrongly identified segmentation points were detected and corrected using articulatory feature based phoneme language model. The segmented signals were mapped to their phonemes. The ambiguities in the recognized phonemes were reduced by using inter and intra word based language models. The recognized phonemes were grouped together to form syllables and then words. The errors in the syllables and words were detected and corrected by using the syllable and morpheme based language models developed for Tamil language. The performance of the Tamil speech recognition system was improved by using the language models at different phases of speech recognition. Recognition rate of 74.11% was obtained by applying language models at segmentation phase, which was further improved to 84.11% at phoneme recognition phase and finally to 87.1% at syllable level and word level recognition phase. Thus the use of language models has drastically reduced the error rates at various levels and improved the recognition rate of Tamil speech recognition system. Keywords: Language model, articulatory features, morphemes, syllables.

Investigation and modeling of the structure of texting language

Language usage over computer mediated discourses, such as chats, emails and SMS texts, significantly differs from the standard form of the language and is referred to as texting language (TL). The presence of intentional misspellings significantly decrease the accuracy of existing spell checking techniques for TL words. In this work, we formally investigate the nature and type of compressions used in SMS texts, and develop a Hidden Markov Model based word-model for TL. The model parameters have been estimated through standard machine learning techniques from a word-aligned SMS and standard English parallel corpus. The accuracy of the model in correcting TL words is 57.7%, which is almost a threefold improvement over the performance of Aspell. The use of simple bigram language model results in a 35% reduction of the relative word level error rates.

A Fast Histogram-Based Postprocessor That Improves Posterior Probability Estimates

Although the outputs of neural network classifiers are often considered to be estimates of posterior class probabilities, the literature that assesses the calibration accuracy of these estimates illustrates that practical networks often fall far short of being ideal estimators. The theorems used to justify treating network outputs as good posterior estimates are based on several assumptions: that the network is sufficiently complex to model the posterior distribution accurately, that there are sufficient training data to specify the network, and that the optimization routine is capable of finding the global minimum of the cost function. Any or all of these assumptions may be violated in practice. This article does three things. First, we apply a simple, previously used histogram technique to assess graphically the accuracy of posterior estimates with respect to individual classes. Second, we introduce a simple and fast remapping procedure that transforms network outputs to provide better estimates of posteriors. Third, we use the remapping in a real-world telephone speech recognition system. The remapping results in a 10% reduction of both word-level error rates (from 4.53% to 4.06%) and sentence-level error rates (from 16.38% to 14.69%) on one corpus, and a 29% reduction at sentence-level error (from 6.3% to 4.5%) on another. The remapping required negligible additional overhead (in terms of both parameters and calculations). McNemar's test shows that these levels of improvement are statistically significant.

Analysis of the Spontaneous Writing Errors of Normal and Aphasic Writers

Samples of spontaneous writing were collected from 150 normal subjects, 13 Broca's aphasics, 23 Wernicke's aphasics and 14 conduction aphasics. The errors obtained were classified using a system derived from investigations of slips of the pen in normal subjects. All three aphasic groups made a higher proportion of word-level errors than did the normal writers. Word-level errors tended to be selection errors (substitutions, blends, neologisms, and omissions) rather than movement errors (anticipations, perseverations and reversals). Conduction aphasics showed proportionately more letter movement and fewer letter selection errors than normal writers while Wernicke's and Broca's aphasics showed similar proportions. Graphic and phonetic similarity was no more important a determinant of letter errors in aphasics than in normals. The span of movement errors was particularly restricted in the Broca's and conduction aphasics. Asking patients to write connected text yields insight into the nature of the underlying disorder that could not be obtained from studying the writing of single words.