Chinese Word Segmentation Research Articles

Chinese word segmentation with heterogeneous graph convolutional network

Abstract Recently, deep learning methods have achieved remarkable success in the Chinese word segmentation (CWS) task. Some of them enhance the CWS model by utilizing contextual features and external resources (e.g., sub-words, lexicon, and syntax). However, existing approaches fail to fully use the heterogeneous features and their structural information. Therefore, in this paper, we propose a heterogeneous information learning framework for CWS, named heterogeneous graph neural segmenter (HGNSeg), which exploits heterogeneous features with the graph convolutional networks and the pretrained language model. Experimental results on six benchmark datasets (e.g., SIGHAN 2005 and SIGHAN 2008) confirm that HGNSeg can effectively improve the performance of CWS. Importantly, HGNSeg also demonstrates an excellent ability to alleviate the out-of-vocabulary (OOV) issue in cross-domain scenarios.

The role of semantic information in Chinese word segmentation

ABSTRACT Word segmentation is crucial for reading in Chinese, where the absence of explicit word boundaries poses a distinct challenge. Previous studies in Chinese have examined how lexical and sub-lexical variables affect word segmentation. The present study investigated whether higher-level semantic information affects word segmentation using a primed word segmentation task with Overlapping Ambiguous Strings (OAS). An OAS is a three-character string in Chinese (e.g. ABC [in Latin letters]) where the middle character can constitute a word with both the left (word AB) and right (word BC) characters. The OAS was preceded by a semantic or repetition prime (presented for 42, 83, or 200 ms, across participants), priming either AB or BC. The semantic priming effect occurred at the 200-ms Stimulus Onset Asynchrony (SOA), whereas the repetition priming effect occurred at both 83 and 200-ms SOAs. These findings demonstrate that semantic information can affect word segmentation in Chinese within 200 ms.

The Comprehensive Analysis of the Effect of Chinese Word Segmentation on Fuzzy-Based Classification Algorithms for Agricultural Questions

The Comprehensive Analysis of the Effect of Chinese Word Segmentation on Fuzzy-Based Classification Algorithms for Agricultural Questions

Research on the Value and Language Features of Chinese Language and Literature Texts Based on Text Mining Technology

The linguistic characteristics of a literary work are the way of thinking embodied in the author’s use of language. From the textual value of Chinese language literature, this paper analyzes the spiritual connotation of Chinese language literature from two dimensions: reading and education. Based on the web crawler technology, we obtain the text data of Chinese language literature from three writers, Bajin, Yu Zheng and Qiong Yao, preprocess the data through data cleaning, Chinese word segmentation, de-duplication, etc., and extract the feature values of the text by using the TF-IDF algorithm. Then the text documents are mapped onto vectors using the VSM model, and the parameters of the LDA topic model are estimated by the Gibbs sampling algorithm in order to better obtain the topic changes of the Chinese language literature texts. This paper carries out linguistic feature verification from the lexical and similarity features of Chinese language literary texts. It is found that the difference in lexical density between Ba Jin’s Cold Night and Resting Garden is only 2.1 percentage points, and the frequency of the verb “to say” is 1,213 times and 735 times respectively. The average sentence lengths of Yu Zheng and Qiong Yao fluctuate within the range of [18.49,34.27], and Qiong Yao’s works have a higher thematic concentration than Zheng Zheng’s works. Analyzing the linguistic features of Chinese language literary texts based on text mining techniques helps to understand the authors’ language usage methods and helps to promote innovative expression paths in literary texts.

Incorporating knowledge for joint Chinese word segmentation and part-of-speech tagging with SynSemGCN

PurposeRecent trends have shown the integration of Chinese word segmentation (CWS) and part-of-speech (POS) tagging to enhance syntactic and semantic parsing. However, the potential utility of hierarchical and structural information in these tasks remains underexplored. This study aims to leverage multiple external knowledge sources (e.g. syntactic and semantic features, lexicons) through various modules for the joint task.Design/methodology/approachWe introduce a novel learning framework for the joint CWS and POS tagging task, utilizing graph convolutional networks (GCNs) to encode syntactic structure and semantic features. The framework also incorporates a pre-defined lexicon through a lexicon attention module. We evaluate our model on a range of public corpora, including CTB5, PKU and UD, the novel ZX dataset and the comprehensive CTB9 dataset.FindingsExperimental results on these benchmark corpora demonstrate the effectiveness of our model in improving the performance of the joint task. Notably, we find that syntax information significantly enhances performance, while lexicon information helps mitigate the issue of out-of-vocabulary (OOV) words.Originality/valueThis study introduces a comprehensive approach to the joint CWS and POS tagging task by combining multiple features. Moreover, the proposed framework offers potential adaptability to other sequence labeling tasks, such as named entity recognition (NER).

Research on China's End-of-Life Care Policy from the Perspective of Aging Population: A Textual Analysis of 30 Policies from 2017 to 2022

As the proportion of elderly population increases and the level of social development continues to rise, the issue of end-of-life care for the elderly has gradually gained societal attention. End-of-life care, as an emerging field, relies crucially on optimizing top-level policy design for its healthy development. This study focuses on 30 national-level end-of-life care policy texts from 2017 to 2022. Utilizing analysis tools such as ROST CM 6 software, NLPIR Chinese word segmentation system, and SPSS, the research employs a three-dimensional framework of keyword frequency analysis, semantic network analysis, and "policy tools-policy time-policy intensity" to comprehensively explore the current state of China's end-of-life care policy. The study reveals existing issues in the policy's practical operability, uneven application of policy tools, neglected funding and information technology support, and weak regulatory aspects. It emphasizes the need to prioritize a people-centered approach, enhance the overall balance in policy tools, promote the standardization of end-of-life care, and improve a diversified service system. These findings broaden the research perspective in the field of end-of-life care, providing insights for optimizing policy design and advancing nationwide end-of-life care efforts.

Comparison and analysis of two methods for improving the accuracy of OpenNMT in literary and vernacular Chinese translation

As machine translation advances, challenges persist in achieving high accuracy when translating between Literary and Vernacular Chinese. Literary Chinese is known for its concise writing style, often characterized by monosyllabic words. In contrast, Vernacular Chinese incorporates more polysyllabic words. This study utilizes the OpenNMT system to address these challenges and employs different Classical Chinese word segmentation tools to train 2-layer Long Short-Term Memory and Transformer models. These models are then compared and analyzed to measure the improvement in precision. The research findings reveal that adopting a character-level-based word segmentation method for Classical Chinese, coupled with training the Transformer model using OpenNMT, significantly enhances precision. This outcome validates the current observation that existing Classical Chinese word segmentation methods lack sufficient accuracy, consequently impacting the quality of translations between Literary and Vernacular Chinese. By exploring and investigating these approaches, this study contributes to advancing machine translation techniques for improving accuracy in rendering Literary and Vernacular Chinese translations.

Identification of Safety Risk Factors in Metro Shield Construction

Among the construction methods for subway projects, shield method construction technology has become a more widely used construction method for urban subway construction due to the advantages of a high degree of construction mechanization, low impact of the construction process on the environment, and strong adaptability of the shield machine to the stratum, etc. However, because of the complexity of the surrounding buildings (structures) in the subway construction, coupled with the diversity of the subway shield method construction activities and the uncertainties in the construction environment, to a certain extent, it is determined that the subway construction process is very complicated. The purpose of this study is based on the text mining method, where text is mined and utilized to realize the identification, extraction, and display of safety risk factors. Thus, it guides the safety management on site and provides a basis for knowledge reuse in other metro shield construction projects. Firstly, we analyze the shortcomings of safety risk management in domestic and international metro shield construction via a literature review, especially the utilization of safety risk text data. Secondly, we collect the risk reports submitted by all parties via the “Metro Project Safety Risk Early Warning System”, and manually screen the hidden danger statements with risk characterization to establish a corpus. Thirdly, we use the Jieba word separation package to extract and display the safety risk factors, so as to guide the on-site safety management. Subsequently, with the help of the Jieba word segmentation package for Chinese word segmentation, we develop a professional thesaurus to improve the effect of word segmentation; then, we use the TF-IDF parameter assignment to achieve the structural transformation of the text to extract high-frequency vocabulary; finally, from the high-frequency vocabulary to screen words containing the semantics of the risk to establish the risk of an initial set of words, we use the existing standards and norms to form the collection of safety risk factors of subway shield construction and generate the cloud diagram for visual display.

Research advanced in Chinese word segmentation methods and challenges

Chinese word segmentation refers to the process of dividing a sequence of Chinese characters into individual words. It constitutes a fundamental component of Chinese natural language processing. Due to the intricacies of the Chinese language, Chinese word segmentation has garnered significant attention from researchers. Based on a review of historical literature, segmentation methods can be broadly categorized into rule-based, statistical, semantic-based, and comprehension-based approaches. With the advancement of machine learning, neural networks have emerged as the mainstream algorithm for word segmentation. However, Chinese presents several unique challenges, leading to segmentation results that are less effective compared to morphological analysis in languages like English. Moreover, word segmentation faces new challenges such as dependency on the quality and scale of corpora, as well as domain-specific segmentation in diverse fields. Addressing these emerging challenges will undoubtedly become a focal point in future research endeavors in this field. This review provides a comprehensive summary of existing methods, discusses the current state of Chinese word segmentation, and outlines directions for addressing the evolving complexities in the field. As Chinese language processing continues to advance, finding robust solutions for accurate word segmentation remains a critical area of research.

A Systematic Study on the Dilemma and Innovative Path of Rural Family Education Development in the Context of Deep Learning

Abstract In order to better improve the level and quality of rural family education, this paper adopts text mining technology to interpret the relevant data of rural family education under the background of deep learning. On the basis of deep learning algorithms, web crawler technology is used to obtain rural family education research data. It is found that there are null values and repetitive values in the acquired research data, so it is necessary to further clean the research data. After washing, the combined Bi-LSTM-CRF algorithm is used to process the data for Chinese word segmentation, and at the same time, the stop word list is improved in order to achieve the best word segmentation effect. The LDA thematic model was used to determine the influencing factors of rural family education, and then the linear regression algorithm was used to analyze rural family education empirically. Rural teachers with different education levels were not significantly different in Q1 (1.608), Q2 (0.117), Q3 (2.019), Q4 (1.733), and Q5 (2.042), according to the results. The adjusted R2 is 0.133, indicating a fair fit between the model obtained from the regression analysis and the actual data, and the regression coefficient of family education is 0.411 after eliminating the confounding interference of grade and political appearance. This study purposefully proposes a path for the innovative development of rural family education, which can provide a theoretical reference program for the development and construction of modern rural family education.

Research and Application of Legal Consultation database based on multimedia Network

In order to relieve the pressure of people's appeal for legal inquiry, the research and application of legal consultation database based on multimedia network is proposed. The legal consultation system designed in this essay, combined with the current situation of the legal industry, takes the Chinese question answering system as the prototype, combined with the open source data retrieval project Lucene.net, expands the data storage types. In this essay, the Chinese word segmentation system developed by Chinese Academy of Sciences is integrated into Lucene.Net platform to make up for its word segmentation deficiency. And the mutual information technology is used to make the legal related words with the same meaning be searched first. When extracting answers in Chinese question answering system, there are often "missing" and "wrong" answers. In this essay a method of sentence similarity calculation of questions and answers based on Latent Semantic analysis (LSA) is proposed, which uses spatial vector model as representation method and latent semantic analysis theory. A low-dimensional semantic space is constructed by using the dimension reduction method of singular value decomposition, and the sentence similarity between question and answer is computed on the semantic space. Experimental results show that most retrievals can be completed within 20 seconds, and the average unit retrieval time is kept at the millisecond level. Conclusion: This system has more accurate query accuracy and less running time.

Chinese readers utilize emotion information for word segmentation.

We reported a large-scale Internet-based experiment to investigate the impact of emotion information on Chinese word segmentation, in which participants completed an overlapping ambiguous string (OAS) segmentation task and the Chinese version of Beck Depression Inventory-II in a counterbalanced order. OAS is a three-character string (ABC) in which the middle character can form a distinct word with both the character on its left side (word AB) and the character on its right side (word BC). Participants were presented with isolated OASs and were asked to report the word they identified first. Emotional OAS was constructed by a combination of a neutral word and an emotional word, with the neutral and emotional words sharing character B. We orthogonally manipulated the valence of the emotional words (positive vs. negative) and their position in the OAS (left-side vs. right-side). The results showed that compared with neutral words, both positive and negative words were more likely to be segmented, and this segmentation outcome was not affected by readers with different depression tendencies. These findings suggest that emotion information can influence word segmentation, and that both positive and negative words take precedence over neutral words in the word segmentation process. This study provides a new perspective and evidence to understand the impact of emotion information on word processing.

Improved Word Segmentation System for Chinese Criminal Judgment Documents

ABSTRACT In this paper, a system for automatic word segmentation of Chinese criminal judgment documents is proposed. The system uses a hybrid model composed of fine-tuned BERT (Bidirectional Encoder Representations from Transformers), BiLSTM (Bidirectional Long Short Term Memory) and CRF (Conditional Random Field) for named entity recognition, and introduces a custom dictionary that includes common professional terms in Chinese criminal trial documents, as well as a rule system based on judicial system and litigation procedure related regulations, to further improve the accuracy of word segmentation. BERT uses a deep bidirectional Transformer encoder to pre-train general language representations from large-scale unlabeled text corpora. BiLSTM uses two LSTM networks, one for the forward direction and one for the backward direction, to capture the context from both sides of the input sequence. CRF uses a set of features and weights to define a log-linear distribution over the output sequence. Experimental results show that the proposed system has significantly improved word segmentation accuracy compared to the current commonly used Chinese word segmentation models. In the results of the segmentation of the test data, the F1 scores for jieba, THULAC and the segmentation system proposed in this paper are 85.59%, 87.94% and 94.82%, respectively.

Research on the Influence of Socialist Core Value System Construction on College Students’ Ideological and Political Education in the Context of Deep Learning

Abstract In this paper, the word-level N-gram language model applicable to Chinese word segmentation in ideological and political education is obtained through large-scale corpus training, and the Chinese word segmentation path in ideological and political education is calculated according to the conditional probability formula. On the basis of the deep learning algorithm, the Chinese participle algorithm is optimized by introducing word frequency deviation and sorted word frequency deviation indexes, and the cut score with the largest value of the participle indexes is taken as the final participle result output. Starting from the current situation, we determine the evaluation indexes of ideological and political education. Meanwhile, we complete the research design of the impact of the socialist core value system and use the SPSS22.0 and Origin2019 statistical software to carry out the research and analysis of ideological and political education. The results show that the index values of the Chinese word separation algorithm based on the 4-gram language model have also been improved by 1.15%, which means that there exists the possibility of practical application of the model in Chinese word separation in ideological and political education. The college students’ identification with the socialist core values at the national, social and personal levels are all at a high level, with values of 4.28±0.82, 4.32±0.75 and 4.61±0.62 respectively. This study improves students’ ideological-political literacy and provides references to their healthy growth.

Convolutional neural network for substantiation of children's books, intelligent interactive communication and application analysis of voice question answering

At present, there are many children accompanying robots. A key indicator to measure their performance is convenient and efficient human–computer interaction. The implementation of voice question-answering technology needs to solve three problems: voice recognition, the establishment of a knowledge base, and answer matching. As the front-end data entry, speech recognition accuracy is directly related to the answer-matching effect of the back-end question-answering system. Based on convolutional neural network and intelligent communication technology, this paper analyzes the characteristics of children's reading robot and designs and develops intelligent voice interaction function on the Android platform. The focus of this paper is on Chinese word segmentation and keyword extraction and matching in the interaction function. Further, we designed a simple question-and-answer database for the children's reading robot. In this paper, convolutional neural network connection timing is used as the feature parameter. Through a large number of experiments, the structure of the CNN network is constantly adjusted to minimize the word error rate, optimize the model recognition rate, and better identify users' voice problems.

MCA-NER: Multi-Contextualized Adversarial-Based Attentional Deep Neural Network for Named Entity Recognition

Multi-contextualized representations learning is vital for named entity recognition (NER), which is a fundamental task for effectively extracting structured information from unstructured text, and forming knowledge bases. This task is particularly challenging when dealing with Chinese text given the absence of evident word boundaries. Chinese word segmentation (CWS) can be leveraged to recognize word boundaries, but named entities often encompass multiple segmented words, making it crucial to use boundary information to correctly recognize and distinguish the relationships between these words. In this paper, we propose MCA-NER, a multi-contextualized adversarial-based attentional deep learning approach for Chinese NER, which combines CWS and part-of-speech (POS) tagging information with the classic BiLSTM-CRF NER model, using adversarial multi-task learning. The model incorporates several self-attention components for adversarial and multi-task learning, effectively synthesizing task-specific and common information attribution while improving performance across all three tasks. Experimental results on the three datasets provide compelling evidence that supports the effectiveness and performance of our model.

Adversarial Multi-task Learning for Efficient Chinese Named Entity Recognition

Named entity recognition (NER) is a fundamental task for information extraction applications. NER is challenging because of semantic ambiguities in academic literature, especially for non-Latin languages. Besides word semantic information, recognizing Chinese named entities needs to consider word boundary information, as words contained in Chinese texts are not separated with spaces. Leveraging word boundary information could help to determine entity boundaries and thus improve entity recognition performance. In this article, we propose to combine word boundary information and semantic information for named entity recognition based on multi-task adversarial learning. Specifically, we learn commonly shared boundary information of entities from multiple kinds of tasks, including Chinese word segmentation (CWS), part-of-speech (POS) tagging, and entity recognition, with adversarial learning. We learn task-specific semantic information of words from these tasks and combine the learned boundary information with the semantic information to improve entity recognition with multi-task learning. We then propose a compression method based on improved clustering to accelerate the proposed model. We conduct extensive experiments on four public benchmark datasets and two private datasets, compared with state-of-the-art baseline models, and the experimental results demonstrate that our model achieves considerable performance improvements on various evaluation datasets.

Cross-Modal Impact of Recent Word Encountering Experience

ABSTRACT Purpose This study was designed to distinguish the degree of sharing of representations between different modalities by investigating whether a word encountering experience in one modality impacts word processing in another modality. Method In three experiments, participants experienced some words frequently in the auditory modality (Experiment 1, sample size 30, mean age 23.4, 56.7% female, all participants were native Chinese speakers) or visual modality (Experiment 2, sample size 30, mean age 22.4 years, 63.3% female, all participants were native Chinese speakers) in the training session, and were tested whether the word encountering experience impacts the results of Chinese word segmentation in the visual modality in the test session. In Experiment 3 (sample size 30, mean age 24.6 years, 76.7% female, all participants were native Chinese speakers), we used a within-subjects design, in which each participant received both auditory and visual training tasks. Results The results of the three experiments showed that encountering a word frequently in a short period of time in the auditory modality or visual modality can affect word segmentation in Chinese reading, with a recently experienced word being more likely to be segmented as a word. This effect was long-lasting, as it could still be observed after 7 days. Conclusion The results suggest that the effect of a word encountering experience in listening can be transferred to reading. Thus, word encountering experiences should be stored at a location in the mind that is used for both listening and reading.

Discovery of New Words in Tax-related Fields Based on Word Vector Representation

<p>New words detection, as basic research in natural language processing, has gained extensive concern from academic and business communities. When the existing Chinese word segmentation technology is applied in the specific field of tax-related finance, because it cannot correctly identify new words in the field, it will have an impact on subsequent information extraction and entity recognition. Aiming at the current problems in new word discovery, it proposed a new word detection method using statistical features that are based on the inner measurement and branch entropy and then combined with word vector representation. First, perform word segmentation preprocessing on the corpus, calculate the internal cohesion degree of words through statistics of scattered string mutual information, filter out candidate two-tuples, and then filter and expand the two-tuples; next, it locks the boundaries of new words through calculate the branch entropy. Finally, expand the new vocabulary dictionary according to the cosine similarity principle of word vector representation. The unsupervised neologism discovery proposed in this paper allows for automatic growth of the neologism lexicon, experimental results on large-scale corpus verify the effectiveness of this method.</p> <p> </p>

Psychological Emotion Analysis System for Special Needs Children Based on Neural Network

With the development of modern society and the improvement of living standards, care for special needs children has been increasingly highlighted, and numerous corresponding measures such as welfare homes, special education schools, and youth care centers have emerged. Due to the lack of systematic emotional companionship, the mental health of special needs children are bound to be affected. Nowadays, emotional education, analysis, and evaluation are mostly done by psychologists and emotional analysts, and these measures are unpopular. Therefore, many researchers at home and abroad have focused on the solution of psychological issues and the psychological assessment and emotional analysis of such children in their daily lives. In this paper, a special children’s psychological emotional analysis based on neural network is proposed, where the system sends the voice information to a cloud platform through intelligent wearable devices. To ensure that the data collected are valid, a series of pretreatments such as Chinese word segmentation, de-emphasis, and so on are put into the neural network model. The model is based on the further research of transfer learning and Bi-GRU model, which can meet the needs of Chinese text sentiment analysis. The completion rate of the final model test has reached 97%, which means that it is ready for use. Finally, a web page is designed, which can evaluate and detect abnormal psychological state, and at the same time, a personal emotion database can also be established.