Chinese Natural Language Processing Research Articles

LLaMA-LoRA Neural Prompt Engineering: A Deep Tuning Framework for Automatically Generating Chinese Text Logical Reasoning Thinking Chains

Abstract The exption of Chinese natural language processing (NLP) has stimulated research in the broader NLP domain. However, existing large language models have limitations in comprehending and reasoning in Chinese. This paper addresses these limitations by enhancing Chinese language models comprehension and reasoning capabilities while minimizing resource requirements. We propose LLaMA-LoRA, a neural prompt engineering framework that builds upon the LLaMA-13B model and incorporates the Low-Rank Adaptation(LoRA) of Large Language Models technique for refinement. Chain-of-Thought(CoT) are crucial for generating intermediate reasoning chains in language models, but their effectiveness can be limited by isolated language patterns. Erroneous reasoning resulting from conventional prompts negatively impacts model performance. Automatic prompts are introduced to encourage reasoning chain generation and accurate answer inference. Training the model with an extensive corpus of Chinese CoT data enhances its comprehension and reasoning abilities. The LLaMA-LoRA model demonstrates exceptional performance across numerous Chinese language tasks, surpassing benchmark performance achieved by related language models such as GPT-3.5, Chat-GLM, and OpenAssistant, delivering accurate, comprehensive, and professional answers. The availability of our open-source model code facilitates further research in the field of Chinese text logical reasoning thinking chains.

A Polishing Model for Machine-Generated Ancient Chinese Poetry

Machine poetry generation has been studied for decades, among which ancient Chinese poetry is still challenging in the field of poetry generation due to its unique regularity and rhythm. The quality improvement of ancient Chinese poetries is one of the most promising research areas of ancient Chinese Natural Language Processing. This paper proposes an ancient Chinese poetry polishing model, which is used for polishing to obtain high-quality ancient Chinese poetry. The model consists of a detection network and a correction network. The detection network based on BiLSTM and CRF is used to detect different types of low-quality words in poems. The correction network based on the BERT model is used to modify the detected low-quality words in the global context. The polishing process is iteratively performed until the model judges that there are no low-quality words in the poem. The results show that the polished poems are improved in multiple evaluations. Compared with existing polishing models, the model proposed in this paper performs better in both automatic evaluation and human evaluation when the number of parameters is reduced.

Research advanced in Chinese word segmentation methods and challenges

Chinese word segmentation refers to the process of dividing a sequence of Chinese characters into individual words. It constitutes a fundamental component of Chinese natural language processing. Due to the intricacies of the Chinese language, Chinese word segmentation has garnered significant attention from researchers. Based on a review of historical literature, segmentation methods can be broadly categorized into rule-based, statistical, semantic-based, and comprehension-based approaches. With the advancement of machine learning, neural networks have emerged as the mainstream algorithm for word segmentation. However, Chinese presents several unique challenges, leading to segmentation results that are less effective compared to morphological analysis in languages like English. Moreover, word segmentation faces new challenges such as dependency on the quality and scale of corpora, as well as domain-specific segmentation in diverse fields. Addressing these emerging challenges will undoubtedly become a focal point in future research endeavors in this field. This review provides a comprehensive summary of existing methods, discusses the current state of Chinese word segmentation, and outlines directions for addressing the evolving complexities in the field. As Chinese language processing continues to advance, finding robust solutions for accurate word segmentation remains a critical area of research.

Chinese news topic prediction using bidirectional encoder representation from transformers

Nowadays, there are many researches on natural language processing (NLP). Through the research of NLP method, many problems in machine learning field have been solved. However, since the study of Chinese NLP has not developed rapidly until recent years, there is still much to be studied on Chinese NLP. As an excellent pre-training model, whether Bidirectional Encoder Representation from Transformers (BERT) performs well on specific Chinese NLP remains to be studied. Therefore, this paper uses BERT for Chinese NLP, and trains BERT model by collecting news title data to achieve Chinese text classification. Finally, the prediction results are studied by statistical methods. The research shows that BERT method performs well on Chinese NLP and can predict different types of news headlines well. Although it performs differently on different kinds of titles, its performance is satisfactory on the whole, and the prediction results are relatively balanced in different categories. Therefore, BERT can be used as a very practical and efficient NLP method. At the same time, it can also be predicted that it will play a great role in Chinese NLP.

Text Processing Method based on Natural Language Processing

Today, natural language processing is booming and is widely used in different fields, especially for text processing. In many Chinese natural language processing tasks such as information retrieval, named entity recognition, syntactic analysis, etc., word segmentation and part-of-speech tagging are often the first steps. This paper mainly selects sequence labeling model and BERT pre-training model, specifically studies how to use natural language processing technology for text processing, and analyzes and summarizes the advantages and disadvantages of the two models.

CLART: A cascaded lattice-and-radical transformer network for Chinese medical named entity recognition

Chinese medical named entity recognition (NER) is a fundamental task in Chinese medical natural language processing, aiming to recognize Chinese medical entities within unstructured medical texts. However, it poses significant challenges mainly due to the extensive usage of medical terms in Chinese medical texts. Although previous studies have made attempts to incorporate lexical or radical knowledge in order to improve the comprehension of medical texts, these studies either focus solely on one of these aspects or utilize a basic concatenation operation to combine these features, which fails to fully utilize the potential of lexical and radical knowledge. In this paper, we propose a novel Cascaded LAttice-and-Radical Transformer (CLART) network to exploit both lexical and radical information for Chinese medical NER. Specifically, given a sentence, a medical lexicon, and a radical dictionary, we first construct a flat lattice (i.e., character-word sequence) for the sentence and radical components of each Chinese character through word matching and radical parsing, respectively. We then employ a lattice Transformer module to capture the dense interactions between characters and matched words, facilitating the enhanced utilization of lexical knowledge. Subsequently, we design a radical Transformer module to model the dense interactions between the lattice and radical features, facilitating better fusion of the lexical and radical knowledge. Finally, we feed the updated lattice-and-radical-aware character representations into a Conditional Random Fields (CRF) decoder to obtain the predicted labels. Experimental results conducted on two publicly available Chinese medical NER datasets show the effectiveness of the proposed method.

Automatic Essay Evaluation Technologies in Chinese Writing—A Systematic Literature Review

Automatic essay evaluation, an essential application of natural language processing (NLP) technology in education, has been increasingly employed in writing instruction and language proficiency assessment. Because automatic Chinese Essay Evaluation (ACEE) has made some breakthroughs due to the rapid development of upstream Chinese NLP technology, many evaluation tools have been applied in teaching practice and high-risk evaluation processes. However, the development of ACEE is still in its early stages, with many technical bottlenecks and challenges. This paper systematically explores the current research status of corpus construction, feature engineering, and scoring models in ACEE through literature to provide a technical perspective for stakeholders in the ACEE research field. Literature research has shown that constructing the ACEE public corpus is insufficient and lacks an effective platform to promote the development of ACEE research. Various shallow and deep features can be extracted using statistical and NLP techniques in ACEE. However, there are still substantial limitations in extracting grammatical errors and features related to syntax and traditional Chinese Literary style. For the construction of scoring models, existing studies have shown that traditional machine learning and deep learning methods each have advantages in different corpora and feature selections. The deep learning model, which exhibits strong adaptability and multi-task joint learning potential, has broader development space regarding model scalability.

A Chinese Grammatical Error Correction Model Based On Grammatical Generalization And Parameter Sharing

Abstract Chinese grammatical error correction (CGEC) is a significant challenge in Chinese natural language processing. Deep-learning-based models tend to have tens of millions or even hundreds of millions of parameters since they model the target task as a sequence-to-sequence problem. This may require a vast quantity of annotated corpora for training and parameter tuning. However, there are currently few open-source annotated corpora for the CGEC task; the existing researches mainly concentrate on using data augmentation technology to alleviate the data-hungry problem. In this paper, rather than expanding training data, we propose a competitive CGEC model from a new insight for reducing model parameters. The model contains three main components: a sequence learning module, a grammatical generalization module and a parameter sharing module. Experimental results on two Chinese benchmarks demonstrate that the proposed model could achieve competitive performance over several baselines. Even if the parameter number of our model is reduced by 1/3, it could reach a comparable $F_{0.5}$ value of 30.75%. Furthermore, we utilize English datasets to evaluate the generalization and scalability of the proposed model. This could provide a new feasible research direction for CGEC research.

RegEMR: a natural language processing system to automatically identify premature ovarian decline from Chinese electronic medical records

BackgroundThe ovarian reserve is a reservoir for reproductive potential. In clinical practice, early detection and treatment of premature ovarian decline characterized by abnormal ovarian reserve tests is regarded as a critical measure to prevent infertility. However, the relevant data are typically stored in an unstructured format in a hospital’s electronic medical record (EMR) system, and their retrieval requires tedious manual abstraction by domain experts. Computational tools are therefore needed to reduce the workload.MethodsWe presented RegEMR, an artificial intelligence tool composed of a rule-based natural language processing (NLP) extractor and a knowledge-based disease scoring model, to automatize the screening procedure of premature ovarian decline using Chinese reproductive EMRs. We used regular expressions (REs) as a text mining method and explored whether REs automatically synthesized by the genetic programming-based online platform RegexGenerator + + could be as effective as manually formulated REs. We also investigated how the representativeness of the learning corpus affected the performance of machine-generated REs. Additionally, we translated the clinical diagnostic criteria into a programmable disease diagnostic model for disease scoring and risk stratification. Four hundred outpatient medical records were collected from a Chinese fertility center. Manual review served as the gold standard, and fivefold cross-validation was used for evaluation.ResultsThe overall F-score of manually built REs was 0.9444 (95% CI 0.9373 to 0.9515), with no significant difference (paired t test p > 0.05) compared with machine-generated REs that could be affected by training set sizes and annotation portions. The extractor performed effectively in automatically tracing the dynamic changes in hormone levels (F-score 0.9518–0.9884) and ultrasonographic measures (F-score 0.9472–0.9822). Applying the extracted information to the proposed diagnostic model, the program obtained an accuracy of 0.98 and a sensitivity of 0.93 in risk screening. For each specific disease, the automatic diagnosis in 76% of patients was consistent with that of the clinical diagnosis, and the kappa coefficient was 0.63.ConclusionA Chinese NLP system named RegEMR was developed to automatically identify high risk of early ovarian aging and diagnose related diseases from Chinese reproductive EMRs. We hope that this system can aid EMR-based data collection and clinical decision support in fertility centers.

A weakly supervised method for named entity recognition of Chinese electronic medical records.

The field of Chinese medical natural language processing faces a significant challenge in training accurate entity recognition models due to the limited availability of high-quality labeled data. In response, we propose a joint training model, MCBERT-GCN-CRF, which achieves high performance in identifying medical-related entities in Chinese electronic medical records. Additionally, we introduce CM-NER, a 5-step framework that effectively mitigates the effects of noise in weakly labeled data and establishes a principled connection between supervised and weakly supervised named entity recognition. We demonstrate significant improvements in recall rate and accuracy. Our approach outperforms traditional fully supervised pre-training models and other state-of-the-art methods by suppressing noise in weakly labeled data. Our proposed framework achieves an F1 score of 86.29% on the CCKS-2019 dataset, significantly higher than pre-trained model baselines ranging from 74.17 to 83.06%, and higher than the top-performing named entity recognition supervised learning models in the CCKS-2019 competition. Our results demonstrate the effectiveness of our proposed framework and highlight the potential of leveraging unlabeled data to train accurate models for named entity recognition in Chinese medical natural language processing. This research has significant implications for advancing natural language processing techniques in the medical domain and improving patient care.

Enhancing Chinese Character Representation With Lattice-Aligned Attention.

Word-character lattice models have been proved to be effective for some Chinese natural language processing (NLP) tasks, in which word boundary information is fused into character sequences. However, due to the inherently unidirectional sequential nature, prior approaches have only learned sequential interactions of character-word instances but fail to capture fine-grained correlations in word-character spaces. In this article, we propose a lattice-aligned attention network (LAN) that aims to model dense interactions over word-character lattice structure for enhancing character representations. By carefully combining cross-lattice module, gated word-character semantic fusion unit, and self-lattice attention module, the network can explicitly capture fine-grained correlations across different spaces (e.g., word-to-character and character-to-character), thus significantly improving model performance. Experimental results on three Chinese NLP benchmark tasks demonstrate that LAN obtains state-of-the-art results compared to several competitive approaches.

Generating risk response measures for subway construction by fusion of knowledge and deep learning

This study aims to propose a method to generate response measures for subway construction risks. The HowNet semantic theoretical system was selected and incorporated the knowledge of subway construction risk management to develop the semantic knowledge base. Then, a risk response generation algorithm was designed based on Knowledge-enabled Bidirectional Encoder Representation from Transformers (K-BERT) to fuse knowledge and deep learning (DL). Meanwhile, the BERT in the original K-BERT model was replaced by the Masked Language Model (MLM) as correction BERT (MacBERT) pre-training model suitable for Chinese natural language processing (NLP) tasks. The data-driven and knowledge-driven approaches were combined for better performance of DL. The pre-training model based on MacBERT was selected to undertake the comparative experiment of fusing knowledge and DL. The results showed that the proposed algorithm with the K-BERT fusion of knowledge and DL had better performance than the MacBERT-based model without the knowledge base.

Sentiment annotations for 3827 simplified Chinese characters.

Sentiment analysis in Chinese natural language processing has been largely based on words annotated with sentiment categories or scores. Characters, however, are the basic orthographic, phonological, and in most cases, semantic units in the Chinese language. This study collected sentiment annotations for 3827 characters. The ratings demonstrated high levels of reliability, and were validated through a comparison with the ratings of some characters' word equivalents reported in a previous norming study. Relations with other lexico-semantic variables and character processing efficiency were investigated. Furthermore, analyses of the association between constituent character valence and word valence revealed semantic compositionality and sentiment fusion characteristic of larger Chinese linguistic units. These ratings for characters, expanding current Chinese sentiment lexicons, can be utilized for the purposes of more precise stimuli assessment in research on Chinese character processing and more efficient sentiment analysis equipped with annotations of single-character words.

Tsetlin Machine for Sentiment Analysis and Spam Review Detection in Chinese

In Natural Language Processing (NLP), deep-learning neural networks have superior performance but pose transparency and explainability barriers, due to their black box nature, and, thus, there is lack of trustworthiness. On the other hand, classical machine learning techniques are intuitive and easy to understand but often cannot perform satisfactorily. Fortunately, many research studies have recently indicated that the newly introduced model, Tsetlin Machine (TM), has reliable performance and, at the same time, enjoys human-level interpretability by nature, which is a promising approach to trade off effectiveness and interpretability. However, nearly all of the related works so far have concentrated on the English language, while research on other languages is relatively scarce. So, we propose a novel method, based on the TM model, in which the learning process is transparent and easily-understandable for Chinese NLP tasks. Our model can learn semantic information in the Chinese language by clauses. For evaluation, we conducted experiments in two domains, namely sentiment analysis and spam review detection. The experimental results showed thatm for both domains, our method could provide higher accuracy and a higher F1 score than complex, but non-transparent, deep-learning models, such as BERT and ERINE.

A Data-Driven Model for Automated Chinese Word Segmentation and POS Tagging.

Chinese natural language processing tasks often require the solution of Chinese word segmentation and POS tagging problems. Traditional Chinese word segmentation and POS tagging methods mainly use simple matching algorithms based on lexicons and rules. The simple matching or statistical analysis requires manual word segmentation followed by POS tagging, which leads to the inability to meet the practical requirements for label prediction accuracy. With the continuous development of deep learning technology, data-driven machine learning models provide new opportunities for automated Chinese word segmentation and POS tagging. Therefore, a data-driven automated Chinese word segmentation and POS tagging model is proposed in order to address the above problems. Firstly, the main idea and overall framework of the proposed automated model are outlined, and the tagging strategy and neural network language model used are described. Secondly, two main optimisations are made on the input side of the model: (1) the use of word2Vec for the representation of text features, thus representing the text as a distributed word vector; and (2) the use of an improved AlexNet for efficient encoding of long-range word, and the addition of an attention mechanism to the model. Finally, on the output side, an additional auxiliary loss function was designed to optimise the Chinese text based on its frequency. The experimental results show that the proposed model can significantly improve the accuracy and operational efficiency of Chinese word segmentation and POS tagging compared with other existing models, thus verifying its effectiveness and advancement.

Evaluation of Chinese Natural Language Processing System Based on Metamorphic Testing

A natural language processing system can realize effective communication between human and computer with natural language. Because its evaluation method relies on a large amount of labeled data and human judgment, the question of how to systematically evaluate its quality is still a challenging task. In this article, we use metamorphic testing technology to evaluate natural language processing systems from the user’s perspective to help users better understand the functionalities of these systems and then select the appropriate natural language processing system according to their specific needs. We have defined three metamorphic relation patterns. These metamorphic relation patterns respectively focus on some characteristics of different aspects of natural language processing. Moreover, on this basis, we defined seven metamorphic relations and chose three tasks (text similarity, text summarization, and text classification) to evaluate the quality of the system. Chinese is used as target language. We extended the defined abstract metamorphic relations to these tasks, and seven specific metamorphic relations were generated for each task. Then, we judged whether the metamorphic relations were satisfied for each task, and used them to evaluate the quality and robustness of the natural language processing system without reference output. We further applied the metamorphic test to three mainstream natural language processing systems (including BaiduCloud API, AliCloud API, and TencentCloud API), and on the PWAS-X datasets, LCSTS datasets, and THUCNews datasets. Experiments were carried out, revealing the advantages and disadvantages of each system. These results further show that the metamorphic test can effectively test the natural language processing system without annotated data.

Word‐Based Method for Chinese Part‐of‐Speech via Parallel and Adversarial Network

Chinese part-of-speech (POS) tagging is an essential task for Chinese downstream natural language processing tasks. The accuracy of the Chinese POS task will drop dramatically by word-based methods because of the segmentation errors and the word sparsity. Also, there are several Chinese POS tagging sets with different criteria. Some of them only have a small-scale annotated corpus and are hard to train. To this end, we propose a modified word-based transformer neural network architecture. Meanwhile, we utilize an adversarial transfer learning method that splits the architecture into shared and private parts. This work directly improves the ability of the word-based model, instead of adopting a joint character-based method. Extensive experiments show that our method achieves state-of-the-art performance on all datasets, and more importantly, our method improves performance effectively for the word-based Chinese sequence labeling task.

Expletive ta in Mandarin Chinese: A Quantitative Study

Expletive pronouns have posed significant challenges to both linguistic studies and natural language processing in Chinese. Based on the data from the Chinese corpus at the Center for Chinese Linguistics of Peking University (CCL), this paper investigates the automatic dependency parsing errors of expletive ta (“it/he”) and relative constructions and provides a detailed quantitative analysis of the characteristics of the constructions in question. The findings not only provide evidence to show that even in radical pro-drop languages such as Mandarin, pleonastics can occur in both object and subject positions but also offer important insights for Chinese natural language processing.

StaResGRU-CNN with CMedLMs: A stacked residual GRU-CNN with pre-trained biomedical language models for predictive intelligence

As a task requiring strong professional experience as supports, predictive biomedical intelligence cannot be separated from the support of a large amount of external domain knowledge. By using transfer learning to obtain sufficient prior experience from massive biomedical text data, it is essential to promote the performance of specific downstream predictive and decision-making task models. This is an efficient and convenient method, but it has not been fully developed for Chinese Natural Language Processing (NLP) in the biomedical field. This study proposes a Stacked Residual Gated Recurrent Unit-Convolutional Neural Networks (StaResGRU-CNN) combined with the pre-trained language models (PLMs) for biomedical text-based predictive tasks. Exploring related paradigms in biomedical NLP based on transfer learning of external expert knowledge and comparing some Chinese and English language models. We have identified some key issues that have not been developed or those present difficulties of application in the field of Chinese biomedicine. Therefore, we also propose a series of Chinese bioMedical Language Models (CMedLMs) with detailed evaluations of downstream tasks. By using transfer learning, language models are introduced with prior knowledge to improve the performance of downstream tasks and solve specific predictive NLP tasks related to the Chinese biomedical field to serve the predictive medical system better. Additionally, a free-form text Electronic Medical Record (EMR)-based Disease Diagnosis Prediction task is proposed, which is used in the evaluation of the analyzed language models together with Clinical Named Entity Recognition, Biomedical Text Classification tasks. Our experiments prove that the introduction of biomedical knowledge in the analyzed models significantly improves their performance in the predictive biomedical NLP tasks with different granularity. And our proposed model also achieved competitive performance in these predictive intelligence tasks.

Incorporating multi-level CNN and attention mechanism for Chinese clinical named entity recognition

Named entity recognition (NER) is a fundamental task in Chinese natural language processing (NLP) tasks. Recently, Chinese clinical NER has also attracted continuous research attention because it is an essential preparation for clinical data mining. The prevailing deep learning method for Chinese clinical NER is based on long short-term memory (LSTM) network. However, the recurrent structure of LSTM makes it difficult to utilize GPU parallelism which to some extent lowers the efficiency of models. Besides, when the sentence is long, LSTM can hardly capture global context information. To address these issues, we propose a novel and efficient model completely based on convolutional neural network (CNN) which can fully utilize GPU parallelism to improve model efficiency. Moreover, we construct multi-level CNN to capture short-term and long-term context information. We also design a simple attention mechanism to obtain global context information which is conductive to improving model performance in sequence labeling tasks. Besides, a data augmentation method is proposed to expand the data volume and try to explore more semantic information. Extensive experiments show that our model achieves competitive performance with higher efficiency compared with other remarkable clinical NER models.