Word Segmentation Research Articles

A Comparative Study on Selecting Acoustic Modeling Units for WFST-based Mongolian Speech Recognition

Traditional weighted finite-state transducer– (WFST) based Mongolian automatic speech recognition (ASR) systems use phonemes as pronunciation lexicon modeling units. However, Mongolian is an agglutinative, low-resource language, and building an ASR system based on the phoneme pronunciation lexicon remains a challenge for various reasons. First, the phoneme pronunciation lexicon manually constructed by Mongolian linguists is finite, which is usually used to build a grapheme-to-phoneme conversion (G2P) model to frequently expand new words. However, the data sparsity decreases the robustness of the G2P model and affects the performance of the final ASR system. Second, homophones and polysyllabic words are common in Mongolian, which has a certain impact on the construction of the Mongolian acoustic model. To address these problems, in this work, we first propose a grapheme-to-phoneme alignment model to obtain the mapping relationship between phonemes and subword units. Then, we construct an acoustic subword segmentation set to segment words directly instead of using the traditional G2P method to predict phoneme sequences to expand the pronunciation lexicon. Further, by analyzing the Mongolian encoding form, we also propose an acoustic subword modeling units construction method that removes control characters. Finally, we investigate various acoustic subword modeling units for pronunciation lexicon construction for the Mongolian ASR system. Experiments on a Mongolian dataset with 325 hours of training show that the pronunciation lexicon based on the acoustic subword modeling unit can effectively construct the WFST-based Mongolian ASR system. Further, removing the control characters when building the acoustic subword modeling unit can further improve the ASR system performance.

Text Processing Method based on Natural Language Processing

Today, natural language processing is booming and is widely used in different fields, especially for text processing. In many Chinese natural language processing tasks such as information retrieval, named entity recognition, syntactic analysis, etc., word segmentation and part-of-speech tagging are often the first steps. This paper mainly selects sequence labeling model and BERT pre-training model, specifically studies how to use natural language processing technology for text processing, and analyzes and summarizes the advantages and disadvantages of the two models.

Research on performance variations of classifiers with the influence of pre-processing methods for Chinese short text classification.

Text pre-processing is an important component of a Chinese text classification. At present, however, most of the studies on this topic focus on exploring the influence of preprocessing methods on a few text classification algorithms using English text. In this paper we experimentally compared fifteen commonly used classifiers on two Chinese datasets using three widely used Chinese preprocessing methods that include word segmentation, Chinese specific stop word removal, and Chinese specific symbol removal. We then explored the influence of the preprocessing methods on the final classifications according to various conditions such as classification evaluation, combination style, and classifier selection. Finally, we conducted a battery of various additional experiments, and found that most of the classifiers improved in performance after proper preprocessing was applied. Our general conclusion is that the systematic use of preprocessing methods can have a positive impact on the classification of Chinese short text, using classification evaluation such as macro-F1, combination of preprocessing methods such as word segmentation, Chinese specific stop word and symbol removal, and classifier selection such as machine and deep learning models. We find that the best macro-f1s for categorizing text for the two datasets are 92.13% and 91.99%, which represent improvements of 0.3% and 2%, respectively over the compared baselines.

Convolutional neural network for substantiation of children's books, intelligent interactive communication and application analysis of voice question answering

At present, there are many children accompanying robots. A key indicator to measure their performance is convenient and efficient human–computer interaction. The implementation of voice question-answering technology needs to solve three problems: voice recognition, the establishment of a knowledge base, and answer matching. As the front-end data entry, speech recognition accuracy is directly related to the answer-matching effect of the back-end question-answering system. Based on convolutional neural network and intelligent communication technology, this paper analyzes the characteristics of children's reading robot and designs and develops intelligent voice interaction function on the Android platform. The focus of this paper is on Chinese word segmentation and keyword extraction and matching in the interaction function. Further, we designed a simple question-and-answer database for the children's reading robot. In this paper, convolutional neural network connection timing is used as the feature parameter. Through a large number of experiments, the structure of the CNN network is constantly adjusted to minimize the word error rate, optimize the model recognition rate, and better identify users' voice problems.

Semi-supervised semantic segmentation using cross-consistency training for pavement crack detection

Crack detection is crucial to providing information for assessing pavement condition and maintaining the safety of infrastructure. Traditional manual crack detection methods are relatively time-consuming and thus have gradually been replaced by automatic detection in recent years. However, most semantic segmentation algorithms require the manual labelling of a large amount of data, which consumes a great deal of time. A semi-supervised semantic word segmentation method based on cross-consistency training for road crack detection is established in this paper. To take advantage of unlabelled crack samples, this study enforced consistency between the primary and secondary decoder predictions, using different disturbed versions of the encoder output as inputs to improve the encoder representation. When only 60% of the annotated data were used, the applied method achieved a better performance than other mainstream semantic segmentation algorithms. Therefore, the applied method can be used in automated and efficient pavement detection for pavement inspection and maintenance.

Emotion detection on social media status in Myanmar language

<span lang="EN-US">Many social media emerged and provided services during these years. Most people, especially in Myanmar, use them to express their emotions or moods, learn subjects, sell products, read up-to-date news, and communicate with each other. Emotion detection on social users makes critical tasks in the opinion mining and sentiment analysis. This paper presents the emotion detection system on social media (Facebook) user status or post written in Myanmar (Burmese) language. Before the emotion detection process, the user posts are pre-processed under segmentation, stemming, part-of-speech (POS) tagging, and stop word removal. The system then uses our preconstructed Myanmar word-emotion Lexicon, M-Lexicon, to extract the emotion words from the segmented POS post. The system provides six types of emotion such as surprise, disgust, fear, anger, sadness, and happiness. The system applies naïve Bayes (NB) emotion classifier to examine the emotion in the case of more than two words with different emotion values are extracted. The classifiers also classify the emotion of the users on their posts. The experiment shows that the system can detect 85% accuracy in NB based emotion detection while 86% in recurrent neural network (RNN).</span>

Morpheme-Based Neural Machine Translation Models for Low-Resource Fusion Languages

Neural approaches, which are currently state-of-the-art in many areas, have contributed significantly to the exciting advancements in machine translation. However, Neural Machine Translation (NMT) requires a substantial quantity and good quality parallel training data to train the best model. A large amount of training data, in turn, increases the underlying vocabulary exponentially. Therefore, several proposed methods have been devised for relatively limited vocabulary due to constraints of computing resources such as system memory. Encoding words as sequences of subword units for so-called open-vocabulary translation is an effective strategy for solving this problem. However, the conventional methods for splitting words into subwords focus on statistics-based approaches that mainly conform to agglutinative languages. In these languages, the morphemes have relatively clean boundaries. These methods still need to be thoroughly investigated for their applicability to fusion languages, which is the main focus of this article. Phonological and orthographic processes alter the borders of constituent morphemes of a word in fusion languages. Therefore, it makes it difficult to distinguish the actual morphemes that carry syntactic or semantic information from the word’s surface form, the form of the word as it appears in the text. We, thus, resorted to a word segmentation method that segments words by restoring the altered morphemes. We also compared conventional and morpheme-based NMT subword models. We could prove that morpheme-based models outperform conventional subword models on a benchmark dataset.

Hot Topic Recognition of Health Rumors Based on Anti-Rumor Articles on the WeChat Official Account Platform: Topic Modeling.

Social networks have become one of the main channels for obtaining health information. However, they have also become a source of health-related misinformation, which seriously threatens the public's physical and mental health. Governance of health-related misinformation can be implemented through topic identification of rumors on social networks. However, little attention has been paid to studying the types and routes of dissemination of health rumors on the internet, especially rumors regarding health-related information in Chinese social media. This study aims to explore the types of health-related misinformation favored by WeChat public platform users and their prevalence trends and to analyze the modeling results of the text by using the Latent Dirichlet Allocation model. We used a web crawler tool to capture health rumor-dispelling articles on WeChat rumor-dispelling public accounts. We collected information from health-debunking articles posted between January 1, 2016, and August 31, 2022. Following word segmentation of the collected text, a document topic generation model called Latent Dirichlet Allocation was used to identify and generalize the most common topics. The proportion distribution of the themes was calculated, and the negative impact of various health rumors in different periods was analyzed. Additionally, the prevalence of health rumors was analyzed by the number of health rumors generated at each time point. We collected 9366 rumor-refuting articles from January 1, 2016, to August 31, 2022, from WeChat official accounts. Through topic modeling, we divided the health rumors into 8 topics, that is, rumors on prevention and treatment of infectious diseases (1284/9366, 13.71%), disease therapy and its effects (1037/9366, 11.07%), food safety (1243/9366, 13.27%), cancer and its causes (946/9366, 10.10%), regimen and disease (1540/9366, 16.44%), transmission (914/9366, 9.76%), healthy diet (1068/9366, 11.40%), and nutrition and health (1334/9366, 14.24%). Furthermore, we summarized the 8 topics under 4 themes, that is, public health, disease, diet and health, and spread of rumors. Our study shows that topic modeling can provide analysis and insights into health rumor governance. The rumor development trends showed that most rumors were on public health, disease, and diet and health problems. Governments still need to implement relevant and comprehensive rumor management strategies based on the rumors prevalent in their countries and formulate appropriate policies. Apart from regulating the content disseminated on social media platforms, the national quality of health education should also be improved. Governance of social networks should be clearly implemented, as these rapidly developed platforms come with privacy issues. Both disseminators and receivers of information should ensure a realistic attitude and disseminate health information correctly. In addition, we recommend that sentiment analysis-related studies be conducted to verify the impact of health rumor-related topics.

MCA-NER: Multi-Contextualized Adversarial-Based Attentional Deep Neural Network for Named Entity Recognition

Multi-contextualized representations learning is vital for named entity recognition (NER), which is a fundamental task for effectively extracting structured information from unstructured text, and forming knowledge bases. This task is particularly challenging when dealing with Chinese text given the absence of evident word boundaries. Chinese word segmentation (CWS) can be leveraged to recognize word boundaries, but named entities often encompass multiple segmented words, making it crucial to use boundary information to correctly recognize and distinguish the relationships between these words. In this paper, we propose MCA-NER, a multi-contextualized adversarial-based attentional deep learning approach for Chinese NER, which combines CWS and part-of-speech (POS) tagging information with the classic BiLSTM-CRF NER model, using adversarial multi-task learning. The model incorporates several self-attention components for adversarial and multi-task learning, effectively synthesizing task-specific and common information attribution while improving performance across all three tasks. Experimental results on the three datasets provide compelling evidence that supports the effectiveness and performance of our model.

Feature selection algorithm for substation main equipment defect text mining based on natural language processing

AbstractThe dimension of relevant text feature space and feature weight of substation main equipment defect information is high, so it is difficult to accurately select mining features. The Natural Language Processing (NLP) medium and short‐term neural network model is used to realise the defect information text feature word segmentation in the log. After extracting the text features of defect information of main substation equipment with high categories to form the feature space; the TF‐IDF algorithm is designed to calculate the importance weight of text keywords, judge the criticality of defect information text feature vocabulary, accurately locate defect information text features, and realise defect information text feature mining. Experiments show that the algorithm has high precision for specific word segmentation of massive substation main equipment log information.

Perceptual intake explains variability in statistical word segmentation

One of the first problems in language learning is to segment words from continuous speech. Both prosodic and distributional information can be useful, and it is an important question how the two types of information are integrated. In this paper, we propose that the distinction between input (the statistical properties of the syllable sequence), and intake (how learners perceptually represent the syllable sequence) is a useful framework to integrate different sources of information. We took a novel approach, observing how a large number of syllable sequences were segmented. These sequences had the same transitional probability information for finding word boundaries but different syllables in them. We found large variability in the performance of the segmentation task, suggesting that factors other than the statistical properties of sequences were at play. This variability was explored using the input/intake asymmetry framework, which predicted that factors that shaped the representation of different syllable sequences could explain the variability of learning. We examined two factors, the saliency of the rhythm in these syllable sequences and how familiar the novel word forms in the sequence were to the existing lexicon. Both factors explained the variance in the learnability of different sequences, suggesting that processing of the sequences shaped learning. The implications of these results to computational models of statistical learning and broader implications to language learning were discussed.

Word segmentation from transcriptions of child-directed speech using lexical and sub-lexical cues.

We compare two frameworks for the segmentation of words in child-directed speech, PHOCUS and MULTICUE. PHOCUS is driven by lexical recognition, whereas MULTICUE combines sub-lexical properties to make boundary decisions, representing differing views of speech processing. We replicate these frameworks, perform novel benchmarking and confirm that both achieve competitive results. We develop a new framework for segmentation, the DYnamic Programming MULTIple-cue framework (DYMULTI), which combines the strengths of PHOCUS and MULTICUE by considering both sub-lexical and lexical cues when making boundary decisions. DYMULTI achieves state-of-the-art results and outperforms PHOCUS and MULTICUE on 15 of 26 languages in a cross-lingual experiment. As a model built on psycholinguistic principles, this validates DYMULTI as a robust model for speech segmentation and a contribution to the understanding of language acquisition.

Ranjana Script Handwritten Character Recognition using CNN

This paper proposes a public image database for Ranjana script Handwritten Character Datasets (RHCD), publicly available for Ranjana script researchers or anyone interested in the subject. To the best of our knowledge, the Ranjana script Handwritten Character Dataset (RHCD) is the first publicly available database for Ranjana script researchers. Ranjana script descended from the Brahmi script, consists of 36 consonant letters, 16 vowel letters, and 10 numerical letters. The focus of this research is three-fold: the first is to create a new database for Ranjana script Handwritten Character Recognition; the second is to test the character recognition accuracy of the created RHCD using existing CNN algorithms like LeNET-5, AlexNET, and ZFNET algorithm; the third is to propose a model by investigating different hyper-tuning parameters to improve the recognition accuracy of the created RHCD. The research method applied in this study is dataset collection, digitization & cropping, pre-processing, dataset splitting, data augmentation, and finally, implementing the CNN model (existing and proposed). Performance evaluation is based on the test accuracy, precision, recall, and F1-score. The experiment result shows that our model ranks first, with a testing accuracy of 99.73% for 64x64 pixels resolution with precision, recall, and F1-score value 1. Creation and recognition of Ranjana script characters, vowel modifiers, and compound characters can be the next milestone to be achieved. Segmentation of words and sentences into characters and recognizing each character individually can be the next research domain.

Research on named entity recognition of Traditional Chinese Medicine chest discomfort cases incorporating domain vocabulary features

Objective:To promote research on knowledge extraction and knowledge graph construction of chest discomfort medical cases in Traditional Chinese Medicine (TCM), this paper focuses on their named entity recognition (NER). The recognition task includes six entity types: “syndrome”, “symptom”, “etiology and pathogenesis”, “treatment method”, “medication”, and “prescription”. Methods:We annotated data in a BIO (B-begin, I-inside, O-outside) manner. For the characteristics of medical case texts, we proposed a custom dictionary method that can be dynamically updated for word segmentation. To compare the effect of the method on the experimental results, we applied the method in the BiLSTM-CRF model and IDCNN-CRF model, respectively. Results:The models using custom dictionaries (BiLSTM-CRF-Loaded and IDCNN-CRF-Loaded) outperformed the models without custom dictionaries (BiLSTM-CRF and IDCNN-CRF) in accuracy, precision, recall, and F1 score. The BiLSTM-CRF-Loaded model yielded F1 scores of 92.59% and 93.23% on the test set and validation set, respectively, surpassing the BERT-BiLSTM-CRF model by 3.59% and 4.87%. Furthermore, when analyzing the results for the six entity categories separately, we found that the use of custom dictionaries has a marked impact, with the categories of “etiology and pathogenesis” and “syndrome” demonstrating the most noticeable improvements. By comparing the F1 scores, we observed that the entity category “medication” yielded the highest performance, reaching F1 scores of 96.04% and 96.48% on the test set and validation set, respectively. Conclusion:We propose a word segmentation method based on a dynamically updated custom dictionary. The method is combined with the BILSTM-CRF and the IDCNN-CRF models, which enhances the model to recognize domain-specific terms and new entities. It can be widely applied in dealing with complex text structures and texts containing domain-specific terms.

Corporate digital transformation and labor structure upgrading

Corporate digital transformation aims to transform general-purpose technology into specific technology truly needed for development. Unlike informatisation related to business outsourcing, digitalisation needs professional labourers skilled in digital hardware and software, which means that the corporate digital transformation would promote the upgrade of the labour structure. This paper examines the financial and text data of annual reports of listed companies in China to test this theoretical hypothesis. We use word segmentation technology and word frequency statistical methods to construct corporate digital transformation indicators and empirically test corporate digital transformation's impact on labour force structure. The empirical outcomes indicate that corporate digital transformation has significantly increased the proportion of highly educated, highly skilled and research and development researchers. In this process, China's unique ‘engineer dividend’ can enhance the positive impact of digital transformation on upgrading the labour structure. The conclusion of this paper provides rich empirical evidence for accelerating the digital transformation and taking advantage of the engineer dividend in China.

Phrase parsing in a second language as indexed by the closure positive shift: The impact of language experience and acoustic cue salience.

Despite the importance of prosodic processing in utterance parsing, a majority of studies investigating boundary localization in a second language focus on word segmentation. The goal of the present study was to investigate the parsing of phrase boundaries in first and second languages from different prosodic typologies (stress-timed vs. syllable-timed). Fifty English-French bilingual adults who varied in native language (French or English) and second language proficiency listened to English and French utterances with different prosodic structures while event-related brain potentials were recorded. The utterances were built around target words presented either in phrase-final position (bearing phrase-final lengthening) or in penultimate position. Each participant listened to both English and French stimuli, providing data in their native language (used as reference) and their second language. Target words in phrase-final position elicited closure positive shifts across listeners in both languages, regardless of the language-specific acoustic cues associated with phrase-final lengthening (shorter phrase-final lengthening in English compared to French). Interestingly, directional effects were observed, where learning to parse English as a second language in a native-like manner seemed to require a higher proficiency level than learning to parse French as a second language. This pattern of results supports the idea that L2 listeners need to learn to recognize L2-specific phrase-final lengthening regardless of the apparent similarity across languages and that some language combinations might present greater challenges than others.

A traffic anomaly detection approach based on unsupervised learning for industrial cyber–physical system

In Industrial Cyber–Physical Systems (ICPSs), the attacker can intrude into the cyber system through many penetration tools and attack the physical system. Payload-based traffic anomaly detection is a popular technique against these attacks. Due to the imbalanced distribution of normal and attack samples in ICPS, existing payload-based detection methods are mostly implemented based on unsupervised learning, typically comprising a word segmentation model and an unsupervised classifier. However, existing methods may disrupt semantic correlations and face challenges in extracting complex payload dependence relationships. To address these issues, this paper proposes a traffic anomaly detection approach, which consists of a data preprocessing model, an unsupervised word segmentation model, and an unsupervised classification model based on autoencoder. The unsupervised word segmentation model utilizes Long Short-Term Memory (LSTM) to calculate the probability of each word segmentation combination, effectively addressing the issue of inaccurate segmentation results in existing payload segmentation models. The unsupervised classification model, which combines 1D-Convolutional Neural Network (1D-CNN) and Bidirectional Encoder Representation from Transformers (BERT), addresses the challenge of extracting complex payload dependence relationships in existing classification models. The proposed detection approach is evaluated using a Cyber–Physical Attack Dataset (CPAD). Compared with the state-of-the-art detection approaches, the proposed approach has shown a significant improvement in Precision, with an increase of 18.83%. Additionally, the Recall has also been substantially enhanced, with a gain of 22.3%. Overall, the F1 has demonstrated a comprehensive improvement of 20.60%.

Word and character segmentation in ancient handwritten documents in Devanagari and Maithili scripts using horizontal zoning

Word and character segmentation in ancient handwritten documents in Devanagari and Maithili scripts using horizontal zoning

Retracted: The Application of Unsupervised Learning TF-IDF Algorithm in Word Segmentation of Ideological and Political Education

Retracted: The Application of Unsupervised Learning TF-IDF Algorithm in Word Segmentation of Ideological and Political Education

Online Reviews Sentiment Analysis and Product Feature Improvement with Deep Learning

The text mining of online reviews is currently a popular research direction of e-commerce and is considered the next blue ocean. Online reviews can dig out consumer preferences and provide theoretical guidance for the improvement of product features. However, current research mostly focuses on sentiment analysis methods and rarely involves feature extraction and large-scale data recognition. This article uses word segmentation technology to create a new feature extraction method. With the long short-term memory neural network and latent Dirichlet allocation topic model, we propose a product feature improvement model—CESC (Consumer online reviews–Extract short text–Sentiment analysis–Cluster feature). The model can derive the product features and attitudes that consumers prefer based on consumer online reviews and use it to improve product features. According to the experimental results of three electronic products sold on the e-commerce platform, the model can effectively dig out consumer preferences for online reviews. Enterprises can improve the quality of products and services, better meet the needs of consumers, promote consumers’ consumption, and achieve the enterprises’ goals and values.