Chinese Word Embedding Research Articles

A novel few-shot learning framework for rock images dually driven by data and knowledge

In the field of geosciences, the integration of artificial intelligence is transitioning from perceptual intelligence to cognitive intelligence. The simultaneous utilization of knowledge and data in the geoscience domain is a universally addressed concern. In this paper, based on the interpretability of deep learning models for rock images, rock features such as structure, texture, mineral and macroscopic identification characteristics were selected to extract a rock identification subgraph from the petrographic knowledge graph and carry out rock type similarity reasoning. Comparative experiments were conducted on few-shot learning of rock images under the supervision of rock type similarity knowledge. The results of the few-shot learning comparisons demonstrate that the supervision of rock type similarity knowledge significantly enhances performance. Additionally, rock type similarity knowledge exhibits a marginal effect on improving few-shot learning performance. Given the absence of Chinese word embedding and large-scale Chinese pre-trained language models in the geological domain, graph embedding based on domain-specific knowledge graphs in geosciences can offer computable geoscience knowledge for research dually propelled by data and knowledge.

Stroke-Based Autoencoders: Self-Supervised Learners for Efficient Zero-Shot Chinese Character Recognition

Chinese characters carry a wealth of morphological and semantic information; therefore, zero-shot Chinese character recognition with the morphology of Chinese characters has drawn significant attention. The previous methods are mainly based on radical-level decomposition or stroke-level decomposition, which usually cannot capture adequately the structural and spatial information of Chinese characters. In this paper, we develop a stroke-based autoencoder (SAE), to model the sophisticated morphology of Chinese characters with a self-supervised method. Following its canonical writing order, we first represent a Chinese character as a series of stroke images with a fixed writing order, and then our SAE model is trained to reconstruct this stroke image sequence. This pre-trained SAE model can predict the stroke image series for unseen characters, as long as their strokes or radicals are in the training set. We have designed two contrasting SAE architectures on different forms of stroke images. One is fine-tuned on existing stroke-based method for zero-shot recognition of handwritten Chinese characters, and the other is applied to enrich the Chinese word embeddings from their morphological features. The experimental results validate that after pre-training, our SAE architecture outperforms other existing methods in zero-shot recognition and enhances the representation of Chinese characters with their abundant morphological and semantic information.

Exploring Chinese word embedding with similar context and reinforcement learning

Exploring Chinese word embedding with similar context and reinforcement learning

Learning Chinese word embeddings from semantic and phonetic components

Learning Chinese word embeddings from semantic and phonetic components

Improving Chinese Word Representation Using Four Corners Features

Intuitively, word representation for logographic languages like Chinese can be enhanced by its internal characteristics. Several research endeavors tried to learn Chinese word embeddings with characters, radicals, or subcharacters containing rich semantic information. In this paper, motivated by Four-Corner Method for Character Indexation, we extract features from four corners of characters with important morphological charactertics. Based on the features from four corners, we propose a model to utilize characters and four corner features of words to capture both semantic and morphological information. Moreover, we apply an attention scheme to integrate internal information dynamically, which includes two strategies to assign different weights for elements according to the word frequency. Experimental results on social news corpus and Chinese Wikipedia Dump show exploiting the four corner morphological features is crucial for capturing the meanings of Chinese words. Meanwhile, the results on word analogy, word similarity, and text classification tasks demonstrate that our approach obtains better results than state-of-the-art approaches.

LARQS: An Analogical Reasoning Evaluation Dataset for Legal Word Embedding

Applying natural language processing-related algorithms is currently a popular project in legal applications, for instance, document classification of legal documents, contract review and machine translation. Using the above machine learning algorithms, all need to encode the words in the document in the form of vectors. The word embedding model is a modern distributed word representation approach and the most common unsupervised word encoding method. It facilitates subjecting other algorithms and subsequently performing the downstream tasks of natural language processing vis-à-vis. The most common and practical approach of accuracy evaluation with the word embedding model uses a benchmark set with linguistic rules or the relationship between words to perform analogy reasoning via algebraic calculation. This paper proposes establishing a 1,256 Legal Analogical Reasoning Questions Set (LARQS) from the 2,388 Chinese Codex corpus using five kinds of legal relations, which are then used to evaluate the accuracy of the Chinese word embedding model. Moreover, we discovered that legal relations might be ubiquitous in the word embedding model.

A Core Drug Discovery Framework from Large-Scale Literature for Cold Pathogenic Disease Treatment in Traditional Chinese Medicine.

Cold pathogenic disease is a widespread disease in traditional Chinese medicine, which includes influenza and respiratory infection associated with high incidence and mortality. Discovering effective core drugs in Chinese medicine prescriptions for treating the disease and reducing patients' symptoms has attracted great interest. In this paper, we explore the core drugs for curing various syndromes of cold pathogenic disease from large-scale literature. We propose a core drug discovery framework incorporating word embedding and community detection algorithms, which contains three parts: disease corpus construction, drug network generation, and core drug discovery. First, disease corpus is established by collecting and preprocessing large-scale literature about the Chinese medicine treatment of cold pathogenic disease from China National Knowledge Infrastructure. Second, we adopt the Chinese word embedding model SSP2VEC for mining the drug implication implied in the literature; then, a drug network is established by the semantic similarity among drugs. Third, the community detection method COPRA based on label propagation is adopted to reveal drug communities and identify core drugs in the drug network. We compute the community size, closeness centrality, and degree distributions of the drug network to analyse the patterns of core drugs. We acquire 4681 literature from China national knowledge infrastructure. Twelve significant drug communities are discovered, in which the top-10 drugs in every drug community are recognized as core drugs with high accuracy, and four classical prescriptions for treating different syndromes of cold pathogenic disease are discovered. The proposed framework can identify effective core drugs for curing cold pathogenic disease, and the research can help doctors to verify the compatibility laws of Chinese medicine prescriptions.

Pronunciation-Enhanced Chinese Word Embedding

Chinese word embeddings have recently garnered considerable attention. Chinese characters and their sub-character components, which contain rich semantic information, are incorporated to learn Chinese word embeddings. Chinese characters can represent a combination of meaning, structure, and pronunciation. However, existing embedding learning methods focus on the structure and meaning of Chinese characters. In this study, we aim to develop an embedding learning method that can make complete use of the information represented by Chinese characters, including phonology, morphology, and semantics. Specifically, we propose a pronunciation-enhanced Chinese word embedding learning method, where the pronunciations of context characters and target characters are simultaneously encoded into the embeddings. Evaluation of word similarity, word analogy reasoning, text classification, and sentiment analysis validate the effectiveness of our proposed method.

FSPRM: A Feature Subsequence Based Probability Representation Model for Chinese Word Embedding

Chinese word embedding models capture Chinese semantics based on the character feature of Chinese words and the internal features of Chinese characters such as radical, component, stroke, structure and pinyin. However, some features are overlapping and most methods do not consider their relevance. Meanwhile, they express words as point vectors that cannot better capture different aspect semantics of Chinese words. In this paper, we propose a Feature Subsequence based Probability Representation Model (FSPRM) for learning Chinese word embeddings, in which we first integrate the morphological and phonetic features (stroke, structure and pinyin) of Chinese characters and learn their relevance by designing a feature subsequence to capture relatively comprehensive semantics of Chinese words, then feature probability distribution is proposed for capturing different aspect meanings of Chinese words based on the three internal features and probability representation by estimating its mean as the sum of feature subsequences. Chinese words with similar features may have similar semantics, then we map Chinese words to feature probability distributions and design a similarity-based objective for predicting the contextual words of the target word to learn their semantics. Extensive experiments on word analogy, word similarity, text classification and named entity recognition tasks demonstrate that the proposed method outperforms most state-of-the-art approaches.

A Semantic Analysis and Community Detection-Based Artificial Intelligence Model for Core Herb Discovery from the Literature: Taking Chronic Glomerulonephritis Treatment as a Case Study

The Traditional Chinese Medicine (TCM) formula is the main treatment method of TCM. A formula often contains multiple herbs where core herbs play a critical therapeutic effect for treating diseases. It is of great significance to find out the core herbs in formulae for providing evidences and references for the clinical application of Chinese herbs and formulae. In this paper, we propose a core herb discovery model CHDSC based on semantic analysis and community detection to discover the core herbs for treating a certain disease from large-scale literature, which includes three stages: corpus construction, herb network establishment, and core herb discovery. In CHDSC, two artificial intelligence modules are used, where the Chinese word embedding algorithm ESSP2VEC is designed to analyse the semantics of herbs in Chinese literature based on the stroke, structure, and pinyin features of Chinese characters, and the label propagation-based algorithm LILPA is adopted to detect herb communities and core herbs in the herbal semantic network constructed from large-scale literature. To validate the proposed model, we choose chronic glomerulonephritis (CGN) as an example, search 1126 articles about how to treat CGN in TCM from the China National Knowledge Infrastructure (CNKI), and apply CHDSC to analyse the collected literature. Experimental results reveal that CHDSC discovers three major herb communities and eighteen core herbs for treating different CGN syndromes with high accuracy. The community size, degree, and closeness centrality distributions of the herb network are analysed to mine the laws of core herbs. As a result, we can observe that core herbs mainly exist in the communities with more than 25 herbs. The degree and closeness centrality of core herb nodes concentrate on the range of [15, 40] and [0.25, 0.45], respectively. Thus, semantic analysis and community detection are helpful for mining effective core herbs for treating a certain disease from large-scale literature.

Learning Chinese word representation better by cascade morphological n-gram

Word embedding refers to mapping words or phrases to vectors of real numbers. This is the precondition of text classification, sentiment analysis and text mining in natural language processing using deep neural networks. Taking English as an example, most of current word embedding algorithms obtain the vectors by learning the distribution of word’s prefix, suffix, etyma and the entire word itself. Unlike English, Chinese words are composed of components and strokes. Furthermore, those components and strokes usually hint the meaning of the word. Thus, components and strokes distribution must be fully considered and learnt when one’s doing Chinese word embedding. In this paper, we propose a component-based cascade n-gram (CBC n-gram) model and a stroke-based cascade n-gram (SBC n-gram) model. By overlaying component and stroke n-gram vectors on word vectors, we successfully improve Chinese word embedding so as to preserve as more morphological information as possible at different granularity levels. We evaluate our models on word similarity, word analogy and text classification tasks using wordsim-240, wordsim-296, Chinese word analogy dataset and Fudan Corpus, respectively. Experimental and comparison results show that our models outperform other state-of-the-art methods.

Radical and Stroke-Enhanced Chinese Word Embeddings Based on Neural Networks

The internal structural information of words has proven to be very effective for learning Chinese word embeddings. However, most previous attempts made a single form extraction of internal feature to learn representations, ignoring the comprehensive combination of such information. And they focused only on explicit feature of internal structures, even though these structures still have the implicit semantics of words. In this paper, we propose Radical and Stroke-enhanced Word Embeddings (RSWE), a novel method based on neural networks for learning Chinese word embeddings with joint guidance from semantic and morphological internal information. RSWE enables an embedding model to learn simultaneously from (1) implicit semantic information that is exploited from the radicals, and (2) stroke n-grams information that can be explicitly obtained from Chinese words. In the learning process, RSWE uses stroke n-grams to capture the local structural feature of words, and integrates the implicit information exploited from radicals to enhance the semantic of embeddings. Through this combination procedure, semantics of Chinese words are effectively transferred into the learned embeddings. We evaluate the effectiveness of RSWE on word similarity computation, word analogy reasoning, performance over dimensions, performance over learning corpus size, and named entity recognition tasks, the experimental results show that our model outperforms existing state-of-the-art approaches.

Enabling Intelligent Environment by the Design of Emotionally Aware Virtual Assistant: A Case of Smart Campus

With the advent of the 5G and Artificial Intelligence of Things (AIoT) era, related technologies such as the Internet of Things, big data analysis, cloud applications, and artificial intelligence have brought broad prospects to many application fields, such as smart homes, autonomous vehicles, smart cities, healthcare, and smart campus. At present, most university campus app is presented in the form of static web pages or app menus. This study mainly developed a Deep Neural Network (DNN) based emotionally aware campus virtual assistant. The main contributions of this research are: (1) This study introduces the Chinese Word Embedding to the robot dialogue system, effectively improving dialogue tolerance and semantic interpretation. (2) The traditional method of emotion identification must first tokenize the Chinese sentence, analyze the clauses and part of speech, and capture the emotional keywords before being interpreted by the expert system. Different from the traditional method, this study classifies the input directly through the convolutional neural network after the input sentence is converted into a spectrogram by Fourier Transform. (3) This study is presented in App mode, which is easier to use and economical. (4) This system provides a simple voice response interface, without the need for users to find information in complex web pages or app menus.

Improving clinical named entity recognition in Chinese using the graphical and phonetic feature

BackgroundClinical Named Entity Recognition is to find the name of diseases, body parts and other related terms from the given text. Because Chinese language is quite different with English language, the machine cannot simply get the graphical and phonetic information form Chinese characters. The method for Chinese should be different from that for English. Chinese characters present abundant information with the graphical features, recent research on Chinese word embedding tries to use graphical information as subword. This paper uses both graphical and phonetic features to improve Chinese Clinical Named Entity Recognition based on the presence of phono-semantic characters.MethodsThis paper proposed three different embedding models and tested them on the annotated data. The data have been divided into two sections for exploring the effect of the proportion of phono-semantic characters.ResultsThe model using primary radical and pinyin can improve Clinical Named Entity Recognition in Chinese and get the F-measure of 0.712. More phono-semantic characters does not give a better result.ConclusionsThe paper proves that the use of the combination of graphical and phonetic features can improve the Clinical Named Entity Recognition in Chinese.

Learning chinese word embeddings from character structural information

Word embedding is a basic task in natural language processing area. Unlike English, Chinese subword units, such as characters, radicals, and components, contain rich semantic information which can be used to enhance word embeddings. However, existing methods neglect the semantic contribution of corresponding subword units to the word. In this work, we employ attention mechanism to capture the semantic structure of Chinese words and propose a novel framework, named Attention-based multi-Layer Word Embedding model(ALWE). We also design an asynchronous strategy for updating embedding and attention efficiently. Our model learns to share subword information between distinct words selectively and adaptively. Experimental results on the word similarity, word analogy, and text classification show that the proposed model outperforms all baselines, especially when words do not appear frequently. Qualitative analysis further demonstrates the superiority of ALWE.

Joint Fine-Grained Components Continuously Enhance Chinese Word Embeddings

The most common method of word embedding is to learn word vector representations from context information of large-scale text. However, Chinese words usually consist of characters, subcharacters, and strokes, and each part contains rich semantic information. The quality of Chinese word vectors is related to the accuracy of prediction. Therefore, to obtain high-quality Chinese character embedding, we propose a continuously enhanced word embedding model. The model starts with fine-grained strokes and adjacent stroke information and enhances subcharacter embedding by combining the relationship vector representation between strokes. Similarly, we combine the subcharacter relationship vector and the character relationship vector to learn Chinese word embedding based on the enhanced subcharacter embedding. We construct the underlying stroke n-grams and adjacent stroke n-grams and extract the relationship vector used to enhance the relationship between the components, which can be used to learn Chinese word embedding and improve the accuracy. Finally, we evaluate our model on the word similarity calculations and word reasoning tasks.

Learning Chinese Word Embeddings With Words and Subcharacter N-Grams

Co-occurrence information between words is the basis of training word embeddings; besides, Chinese characters are composed of subcharacters, words made up by the same characters or subcharacters usually have similar semantics, but this internal substructure information is usually neglected in popular models. In this paper, we propose a novel method for learning Chinese word embeddings, which takes full use of external co-occurrence context information and internal substructure information. We represent each word as a bag of subcharacter n-grams, and our model learns the vector representation corresponding to the word and its subcharacter n-grams. The final word embeddings are represented as the sum of these two kinds of vector representation, which makes the learned word embeddings can take into account both the internal structure information and external co-occurrence information possible. The experiments show that our method outperforms state-of-the-art performance on benchmarks.

Cw2vec: Learning Chinese Word Embeddings with Stroke n-gram Information

We propose cw2vec, a novel method for learning Chinese word embeddings. It is based on our observation that exploiting stroke-level information is crucial for improving the learning of Chinese word embeddings. Specifically, we design a minimalist approach to exploit such features, by using stroke n-grams, which capture semantic and morphological level information of Chinese words. Through qualitative analysis, we demonstrate that our model is able to extract semantic information that cannot be captured by existing methods. Empirical results on the word similarity, word analogy, text classification and named entity recognition tasks show that the proposed approach consistently outperforms state-of-the-art approaches such as word-based word2vec and GloVe, character-based CWE, component-based JWE and pixel-based GWE.