Pronunciation Error Detection Research Articles

Retracted: An Automatic Pronunciation Error Detection and Correction Mechanism in English Teaching Based on an Improved Random Forest Model

Retracted: An Automatic Pronunciation Error Detection and Correction Mechanism in English Teaching Based on an Improved Random Forest Model

English

English

Pronunciation error detection model based on feature fusion

Mispronunciation detection and diagnosis (MDD) is a specific speech recognition task that aims to recognize the phoneme sequence produced by a user, compare it with the standard phoneme sequence, and identify the type and location of any mispronunciations. However, the lack of large amounts of phoneme-level annotated data limits the performance improvement of the model. In this paper, we propose a joint training approach, Acoustic Error_Type Linguistic (AEL) that utilizes the error type information, acoustic information, and linguistic information from the annotated data, and achieves feature fusion through multiple attention mechanisms. To address the issue of uneven distribution of phonemes in the MDD data, which can cause the model to make overconfident predictions when using the CTC loss, we propose a new loss function, Focal Attention Loss, to improve the performance of the model, such as F1 score accuracy and other metrics. The proposed method in this paper was evaluated on the TIMIT and L2-Arctic public corpora. In ideal conditions, it was compared with the baseline model CNN-RNN-CTC. The F1 score, diagnostic accuracy, and precision were improved by 31.24%, 16.6%, and 17.35% respectively. Compared to the baseline model, our model reduced the phoneme error rate from 29.55% to 8.49% and showed significant improvements in other metrics. Furthermore, experimental results demonstrated that when we have a model capable of accurately obtaining pronunciation error types, our model can achieve results close to the ideal conditions.

Innovative Design of English Teaching Mode for Water Resources Majors Based on Smoothed Long and Short-Term Memory Network Models

Abstract Because the English vocabulary of water resources majors is difficult to understand and awkward to speak, this paper constructs a teaching model combined with pronunciation error detection. With the help of the feature engineering method, we extracted the pronunciation features of English for water resources majors and then used a smooth LSTM network to learn and recognize the pronunciation features so as to detect the errors in pronunciation habits and complete the correction of students’ professional English pronunciation. The pronunciation error detection rate reflects the effect of error detection, and the feasibility of the teaching model is verified according to the mastery degree of specialized vocabulary. The post-test mean value of the experimental class is 76.18 points, and the mean value of the control class is 71.12 points, which is 5.06 points higher than that of the experimental class. From the results of the independent samples t-test, the Sig value is 0.102, which is greater than 0.05, indicating that there is a significant difference in the post-test scores of the two classes. This study has a significant impact on students’ mastery of English vocabulary related to water resources.

5g web service based on AI speech recognition platform in blended teaching application of English in higher education

Abstract This paper first investigates English teaching using an AI speech recognition platform. In the feature value analysis module, using the hidden Markov model is beneficial to improve the accuracy of MFCC features for parameter speech recognition. Therefore, the hidden Markov model is proposed for English audio recognition. Secondly, according to the recognition platform pronunciation error detection, the speech pronunciation scoring algorithm is proposed, and an example analysis of English blended teaching based on AI speech recognition is conducted. The results show that the blended teaching mode is 0.33 points higher than the traditional teaching mode in the entrance examination, 2.96 points higher in the total score of Grade 4, 2.10 points higher in listening, 0.57 points higher in reading, and 0.31 points higher in writing, and the effect of blended teaching is positively correlated with the students’ entrance English level. This study proposes a blended English teaching strategy that utilizes a speech recognition platform to achieve innovation and diversification of English teaching efforts.

A study on teaching English in higher education based on an improved deep belief network

Abstract This paper combines machine learning with acoustic features to design an automatic pronunciation error correction system. The article first adopts Meier’s inverse spectral coefficients and random forest algorithm to classify and detect learners’ pronunciation errors and clarify learners’ pronunciation problems, from which the MFCC-RF model is proposed. Then, using the feature self-learning capability of deep belief networks and the OneClass idea of SVM, we proposed a DBN-SVM model to overcome the shortcomings of the MFCC-RF model in pronunciation classification and error detection due to unbalanced samples and missing data, which resulted in low error detection rate and poor coverage of error types. By comparing the model’s performance for pronunciation error detection, the DBN-SVM model was more accurate than the other two algorithms in detecting the three error types with a stable accuracy of around 80%. Finally, when the experimental class was taught with the automatic pronunciation error correction system, the experimental class improved by 19.5 points after one semester of study, while the control class only improved by 6.8 points. Hence, the DBN-SVM model-based pronunciation mistake correction system has significantly impacted the speed of change and advancement in English teaching techniques while substantially enhancing the quality of oral pronunciation and learning efficiency of English learners.

MAKHRAJ ‘AIN PRONUNCIATION ERROR DETECTION USING MEL FREQUENCY CEPSTRAL COEFFICIENT AND MODIFIED VGG-16

Based on research conducted by the Institute of Qur'anic Sciences (IIQ) as many as 65% of Muslims in Indonesia are illiterate in the Qur'an. In previous studies, research was conducted on the detection of Arabic word pronunciation errors against non-natives using the Mel Frequency Cepstral Coefficient (MFCC) and Support Vector Machine (SVM) methods with a test result of 54.6%. Due to the low accuracy results in previous studies, this study aims to design and build a system that can correct the accuracy of the pronunciation of makhraj letter ‘ain with the method used is a combination of MFCC and Convolutional Neural Network (CNN) with a vgg-16 structure that has been modified. The dataset used is 1,600 voice recordings divided into two categories of the correct pronunciation of the letter ‘ain and incorrect pronunciation of the letter ‘ain and four variations of pronunciation with different vowels with a total data of 800 records in each category. This study conducted several experiments on variations of the CNN kernel. The results of the training model that produced the best accuracy in all variations were the training model on kernels 16, 32, 64 with a final accuracy rate of 100% for all variations with 96% accuracy validation. In the fathah variation, the validation accuracy is 94%. In the variation of dhommah and the variation of kasrah obtained a validation accuracy of 97%. Therefore, this study succeeded in distinguishing the sound of the pronunciation of the letter ‘ain with different vowels and measuring the accuracy of the pronunciation of the letter ‘ain. Implementing the modified vgg-16 produces high accuracy and validation values for each speech variation during the model train process.

A framework for pronunciation error detection and correction for non-native Arab speakers of English language

This paper examines speakers’ systematic errors while speaking English as a foreign language (EFL) among students in Arab countries with the purpose of automatically recognizing and correcting mispronunciations using speech recognition, phonological features, and machine learning. Accordingly, three main steps are implemented towards this purpose: identifying the most frequently wrongly pronounced phonemes by Arab students, analyzing the systematic errors these students make in doing so, and developing a framework that can aid the detection and correction of these pronunciation errors. The proposed automatic detection and correction framework used the collected and labeled data to construct a customized acoustic model to identify and correct incorrect phonemes. Based on the trained data, the language model is then used to recognize the words. The final step includes construction samples of both correct and incorrect pronunciation in the phonemes model and then using machine learning to identify and correct the errors. The results showed that one of the main causes of such errors was the confusion that leads to wrongly utilizing a given sound in place of another. The automatic framework identified and corrected 98.2% of the errors committed by the students using a decision tree classifier. The decision tree classifier achieved the best recognition results compared to the five classifiers used for this purpose.

Automatic Speech Recognition and Pronunciation Error Detection of Dutch Non-native Speech: cumulating speech resources in a pluricentric language

• Improving the performance of Automatic Speech Recognition (ASR) on learner's speech in a non-dominant variety of a pluricentric language by cumulating speech resources from different varieties of the same language. • Through the transfer learning approach, the knowledge from a dominant variety can be transferred to non-dominant variety and this benefits non-native speech recognition. • Introducing plausible pronunciation errors in a native corpus of the non-dominant variety based on knowledge from the dominant variety to evaluate the performance of Pronunciation Error Detection (PED) algorithms. The shortage of large-scale learners’ speech corpora and precise manual annotations are two major challenges for automatic L2 speech recognition and error detection in L2 speech, especially for non-dominant varieties of pluricentric languages. In these cases, collecting and annotating large non-native (L2 learner) corpora for all language varieties is often unattainable. In this study, we investigated ways of addressing these problems through conventional and transfer learning Deep Neural Network (DNN) based Automatic Speech Recognition (ASR) and ASR-based pronunciation error detection (PED) by cumulating Netherlandic Dutch and Flemish Dutch speech resources. First, we show that for ASR the baseline system can be improved by combining the Netherlandic Dutch and Flemish Dutch datasets. Next, through the knowledge learned from models trained on the Netherlandic Dutch data, the Flemish Dutch learners' ASR model can be further improved. In order to evaluate the performance of the PED algorithms in the absence of learner speech data with pronunciation error annotations, we introduced plausible pronunciation errors in the native corpora based on knowledge from Flemish learner speech, in order to simulate non-native speech errors. For PED we found that the results are much better for a GOP classifier trained on Flemish Dutch data than for one trained on Netherlandic Dutch data. PED produced worse results when the Netherlandic Dutch data were merged with the Flemish Dutch data, while for ASR, lower WERs were attained. Whether adding Netherlandic Dutch data to Flemish Dutch data is beneficial, thus seems to depend on the specific task the data are used for. We discuss these results, compare them to those of related research and suggest avenues for future research.

Computer-assisted pronunciation training—Speech synthesis is almost all you need

The research community has long studied computer-assisted pronunciation training (CAPT) methods in non-native speech. Researchers focused on studying various model architectures, such as Bayesian networks and deep learning methods, as well as on the analysis of different representations of the speech signal. Despite significant progress in recent years, existing CAPT methods are not able to detect pronunciation errors with high accuracy (only 60% precision at 40%–80% recall). One of the key problems is the low availability of mispronounced speech that is needed for the reliable training of pronunciation error detection models. If we had a generative model that could mimic non-native speech and produce any amount of training data, then the task of detecting pronunciation errors would be much easier. We present three innovative techniques based on phoneme-to-phoneme (P2P), text-to-speech (T2S) and speech-to-speech (S2S) conversion to generate correctly pronounced and mispronounced synthetic speech. We show that these techniques not only improve the accuracy of three machine learning models for detecting pronunciation errors, but also help establish a new state-of-the-art in the field. Earlier studies have used simple speech generation techniques such as P2P conversion, but only as an additional mechanism to improve the accuracy of pronunciation error detection. We, on the other hand, consider speech generation to be the first-class method of detecting pronunciation errors. The effectiveness of these techniques is assessed in the tasks of detecting pronunciation and lexical stress errors. Non-native English speech corpora of German, Italian, and Polish speakers are used in the evaluations. The best proposed S2S technique improves the accuracy of detecting pronunciation errors in AUC metric by 41% from 0.528 to 0.749 compared to the state-of-the-art approach.

An Automatic Pronunciation Error Detection and Correction Mechanism in English Teaching Based on an Improved Random Forest Model

Teachers in traditional English classes focus more on writing and grammar instruction, while oral language instruction is neglected. In exam-oriented education, most Chinese students can master English written test skills, but only a few students can communicate effectively in English daily. People are progressively realizing that language is a tool for communication and communication in recent years, as the frequency of international exchanges has increased and that language learning should focus on oral language education. However, there are numerous issues with teaching oral English. When students perform individual oral practice after class, for example, they are unable to determine whether their pronunciation is correct. As a result, a computer-assisted study into automatic pronunciation of spoken English has become a viable solution to these issues. However, the present spoken English pronunciation mistake correction model’s accuracy and stability have not yet reached an optimal level. Based on this background, this work provides an enhanced random forest model and uses it to detect and correct automatic pronunciation errors in English classes. The improved random forest (RF) algorithm is used to classify and detect whether the learner’s pronunciation is correct. Mel cepstral coefficient (MFCC) is used for feature extraction, and principal component analysis (PCA) is used for dimensionality reduction of feature data. The experimental structure demonstrates that by using a combination classification framework based on MFCC, PCA, and RF, the learner’s pronunciation difficulty may be resolved. This allows for different error categories to receive feedback corrections.

English Pronunciation Calibration Model Based on Multimodal Acoustic Sensor

In recent years, with the increasing frequency of international exchanges, people have gradually realized that language is a tool of communication and communication, and language learning should attach importance to oral teaching. However, in traditional classrooms, one of the problems faced by oral teaching is the mismatch of the teacher-student ratio: a teacher has to deal with dozens of students, one-on-one oral teaching and pronunciation guidance is impossible, and it is also affected by the teachers and the environment constraints. Therefore, the research on how to efficiently automate pronunciation training is becoming more and more popular. Many phonemes in English have different facial visual features, especially vowels. Almost all of them can be distinguished by the roundness and tightness of the lips in appearance. In order to give full play to the role of lip features in oral pronunciation error detection, this paper proposes a multimodal feature fusion model based on lip angle features. The model interpolates the lip features constructed based on the opening and closing angles and combines audio and video in time series. Feature alignment and fusion and feature learning and classification are realized through the two-way LSTM SOFTMAX layer, and finally, end-to-end pronunciation error detection is realized through CTC. It is verified on the GRID audio and video corpus after phoneme conversion and the self-built multimodal test set. The experimental results show that the model has a higher false pronunciation recognition rate than the traditional single-modal acoustic error detection model. The increase in error detection rate is more obvious. Verification by the audio and video corpus with white noise was added, and the proposed model has better noise immunity than the traditional acoustic model.

An English Pronunciation Quality Evaluation Model Based on Multi-dimensional Features

As a universal language, English has been paid more and more attention, among which oral English learning is very important. In this paper, the two key technologies of pronunciation error detection and quality evaluation are studied, both of them are effectively integrated, aiming to build a model for L2 learners’ English pronunciation quality evaluation. This paper mainly studies two different methods of pronunciation error detection. Based on the speech recognition framework, the standard score is compared with the threshold to judge the correctness of phoneme pronunciation, and the phoneme-dependent threshold is set to improve the maximum Precision to 0.44. By judging the correct pronunciation and confusing phoneme, the accuracy of pronunciation error detection is improved to 81.26%. This paper proposes the fusion algorithm from multi-dimensions of speech fluency and intonation respectively, and a newly designed feature called word duration ratio, which significantly improve the correlation of pronunciation quality evaluation to 0.746.

A Transformation of Oral English Teaching Method in the Network Environment

In order to effectually improve the application effect of the existing oral English teaching methods, this paper studies the conversion of effective oral English teaching methods under the network environment, aiming at the application problems, such as the flexibility of the current education platform. We build an effective oral English teaching platform which is comprised of oral English perception processing module, oral English pronunciation acquisition program module, oral English pronunciation preprocessing module, and oral English pronunciation error detection modeling module. We establish the information technology teaching objectives of the effective oral English teaching model on this basis. This research work is focused on the study of the output driven input-facilitated college oral English teaching model in oral English classroom. It is performed to realize the transformation of effective oral English teaching methods under the network environment. The experimental results show that the transformation effect of the effective teaching method of oral English, under the designed network environment, is better as compared to the existing work. The error detection rate in the proposed methodology reaches up to 98.6%, which is significantly higher than the results obtained by researchers in their existing work. The proposed method has obvious application advantages.

English Pronunciation Standards Based on Multimodal Acoustic Sensors

In recent years, economic globalization is the trend, and communication between countries is getting closer and closer; more and more people begin to pay attention to learning spoken English. The development of computer-aided language learning makes it more convenient for people to learn spoken English; however, the detection and correction of incorrect English pronunciation, which is its core, are still inadequate. In this paper, we propose a multimodal end-to-end English pronunciation error detection and correction model based on audio and video, which does not require phoneme forced alignment of the English pronunciation video signal to be processed, and uses rich audio and video features for English pronunciation error detection, which improves the error detection accuracy to a great extent especially in noisy environments. To address the shortcomings of the current lip feature extraction algorithm which is too complicated and not enough characterization ability, a feature extraction scheme based on the lip opening and closing angle is proposed. The lip syllable frames are obtained by video frame splitting, the syllables are denoised, the key point information of the lips is obtained using a gradient enhancement-based regression tree algorithm, the effects of speaker tilt and movement are removed by scale normalization, and finally, the lip opening and closing angles are calculated using mathematical geometry, and the lip feature values are generated by combining the angle changes.

Analysis of Japanese Expressions and Semantics Based on Link Sequence Classification

Based on locness corpus, this paper uses Wordsmith 6.0, SPSS 24, and other software to explore the use of temporal connectives in Japanese writing by Chinese Japanese learners. This paper proposes a method of tense classification based on the Japanese dependency structure. This method analyzes the results of the syntactic analysis of Japanese dependence and combines the tense characteristics of the target language to extract tense-related information and construct a maximum entropy tense classification model. The model can effectively identify the tense, and its classification accuracy shows the effectiveness of the classification method. This paper proposes a temporal feature extraction algorithm oriented to the hierarchical phrase expression model. The end-to-end speech recognition system has become the development trend of large-scale continuous speech recognition because of its simplicity and efficiency. In this paper, the end-to-end technology based on link timing classification is applied to Japanese speech recognition. Taking into account the characteristics of Japanese hiragana, katakana, and Japanese kanji writing forms, through experiments on the Japanese data set, different suggestions are explored. The final effect is better than mainstream speech recognition systems based on hidden Markov models and two-way long and short-term memory networks. This algorithm can extract the temporal characteristics of rules that meet certain conditions while extracting expression rules. These tense characteristics can guide the selection of rules in the expression process, make the expression results more in line with linguistic knowledge, and ensure the choice of relevant vocabulary and the structural ordering of the language. Through the analysis of time series and static information, we combine the time and space dimensions of the network structure. Using connectionist temporal classification (CTC) technology, an end-to-end speech recognition method for pronunciation error detection and diagnosis tasks is established. This method does not require phonemic information nor does it require forced alignment. The extended initials and finals are the error primitives, and 64 types of errors are designed. The experimental results show that the method can effectively detect the wrong pronunciation, the detection accuracy rate is 87.07%, the false rejection rate is 7.83%, and the error rate is 87.07%. The acceptance rate is 25.97%. This method uses network information more comprehensively than traditional methods, and the model is more effective. After detailed experiments, this article evaluates the prediction effect of this method and previous methods on the data set. This method improves the prediction accuracy by about 15% and achieves the expected goal of the work in this paper.

Non-native acoustic modeling for mispronunciation verification based on language adversarial representation learning

Non-native mispronunciation verification is designed to provide feedback to guide language learners to correct their pronunciation errors in their further learning and it plays an important role in the computer-aided pronunciation training (CAPT) system. Most existing approaches focus on establishing the acoustic model directly using non-native corpus thus they are suffering the data sparsity problem due to time-consuming non-native speech data collection and annotation tasks. In this work, to address this problem, we propose a pre-trained approach to utilize the speech data of two native languages (the learner’s native and target languages) for non-native mispronunciation verification. We set up an unsupervised model to extract knowledge from a large scale of unlabeled raw speech of the target language by making predictions about future observations in the speech signal, then the model is trained with language adversarial training using the learner’s native language to align the feature distribution of two languages by confusing a language discriminator. In addition, sinc filter is incorporated at the first convolutional layer to capture the formant-like feature. Formant is relevant to the place and manner of articulation. Therefore, it is useful not only for pronunciation error detection but also for providing instructive feedback. Then the pre-trained model serves as the feature extractor in the downstream mispronunciation verification task. Through the experiments on the Japanese part of the BLCU inter-Chinese speech corpus, the experimental results demonstrate that for the non-native phone recognition and mispronunciation verification tasks (1) the knowledge learned from two native languages speech with the proposed unsupervised approach is useful for these two tasks (2) our proposed language adversarial representation learning is effective to improve the performance (3) formant-like feature can be incorporated by introducing sinc filter to further improve the performance of mispronunciation verification.

Text-conditioned Transformer for automatic pronunciation error detection

Automatic pronunciation error detection (APED) plays an important role in the domain of language learning. As for the previous ASR-based APED methods, the decoded results need to be aligned with the target text so that the errors can be found out. However, since the decoding process and the alignment process are independent, the prior knowledge about the target text is not fully utilized. In this paper, we propose to use the target text as an extra condition for the Transformer backbone to handle the APED task. The proposed method can output the error states with consideration of the relationship between the input speech and the target text in a fully end-to-end fashion. Meanwhile, as the prior target text is used as a condition for the decoder input, the Transformer works in a feed-forward manner instead of autoregressive in the inference stage, which can significantly boost the speed in the actual deployment. We set the ASR-based Transformer as the baseline APED model and conduct several experiments on the L2-Arctic dataset. The results demonstrate that our approach can obtain 8.4% relative improvement on the F1 score metric.

基于迁移学习的端到端发音检错研究

基于迁移学习的端到端发音检错研究

Improving Mispronunciation Detection of Arabic Words for Non-Native Learners Using Deep Convolutional Neural Network Features

Computer-Aided Language Learning (CALL) is growing nowadays because learning new languages is essential for communication with people of different linguistic backgrounds. Mispronunciation detection is an integral part of CALL, which is used for automatic pointing of errors for the non-native speaker. In this paper, we investigated the mispronunciation detection of Arabic words using deep Convolution Neural Network (CNN). For automated pronunciation error detection, we proposed CNN features-based model and extracted features from different layers of Alex Net (layers 6, 7, and 8) to train three machine learning classifiers; K-nearest neighbor (KNN), Support Vector Machine (SVM) and Random Forest (RF). We also used a transfer learning-based model in which feature extraction and classification are performed automatically. To evaluate the performance of the proposed method, a comprehensive evaluation is provided on these methods with a traditional machine learning-based method using Mel Frequency Cepstral Coefficients (MFCC) features. We used the same three classifiers KNN, SVM, and RF in the baseline method for mispronunciation detection. Experimental results show that with handcrafted features, transfer learning-based method and classification based on deep features extracted from Alex Net achieved an average accuracy of 73.67, 85 and 93.20 on Arabic words, respectively. Moreover, these results reveal that the proposed method with feature selection achieved the best average accuracy of 93.20% than all other methods.