Word Segmentation Algorithm Research Articles

Research on sentiment analysis based on the Bilibili video barrage

Based on the analysis of the emotions and mentality of site B users watching videos, this paper proposes a method to visualize important attributes such as high-frequency words, the proportion of positive and negative comments, and word cloud diagrams. In the context of the rise of the Internet and the increasing application of Web 2.0, this paper took the Bilibili barrage text as the research object. Because of the large amount of barrage data, only individual barrages were selected as the analysis objects according to the requirements. The crawler was used to preprocess the crawled video barrage. Then machine translation knowledge and four algorithms such as the word segmentation algorithm and sentiment analysis algorithm were used to analyze the sentiment of the video barrage from three different dimensions and compare the results. Through the analysis of the visualization results, the differences in the emotional distribution of different video barrages were compared, and two important conclusions were drawn: First, the mentality of Bilibili users watching videos is positive; second, there is a certain correlation between the content of the video and the emotional orientation of the barrage, and mutual prediction can be made between the two. However, the research in this paper is only the tip of the iceberg in the research of public opinion analysis. At present, the application of sentiment analysis in public opinion still faces difficulties. How to optimize the algorithm model according to the current situation requires researchers to conduct deeper research and more extensive thinking.

Research advanced in Chinese word segmentation methods and challenges

Chinese word segmentation refers to the process of dividing a sequence of Chinese characters into individual words. It constitutes a fundamental component of Chinese natural language processing. Due to the intricacies of the Chinese language, Chinese word segmentation has garnered significant attention from researchers. Based on a review of historical literature, segmentation methods can be broadly categorized into rule-based, statistical, semantic-based, and comprehension-based approaches. With the advancement of machine learning, neural networks have emerged as the mainstream algorithm for word segmentation. However, Chinese presents several unique challenges, leading to segmentation results that are less effective compared to morphological analysis in languages like English. Moreover, word segmentation faces new challenges such as dependency on the quality and scale of corpora, as well as domain-specific segmentation in diverse fields. Addressing these emerging challenges will undoubtedly become a focal point in future research endeavors in this field. This review provides a comprehensive summary of existing methods, discusses the current state of Chinese word segmentation, and outlines directions for addressing the evolving complexities in the field. As Chinese language processing continues to advance, finding robust solutions for accurate word segmentation remains a critical area of research.

How effective is authoritarian environmentalism？ —evidence from China

Authoritarian environmentalism (AE) in China is embedded in the historical evolution of political centralization and has obvious path dependence; however, its effectiveness lacks empirical evidence. In the light of the cognitive motivation theory and the corpus linguistics, this study uses the multigrain Chinese word segmentation algorithm to construct the will index of provincial and central governments' environmental governance. With this will index and the provincial panel data in China from 2004 to 2017, this paper systematically examines the effectiveness and spatial heterogeneity of AE with the System GMM method. First, there is a negative correlation between the central government's will for environmental governance and environmental quality in provincial administrative regions, and top–down AE, characterized by political centralization, is less effective in China. However, the interaction term coefficients between the central and provincial governments are significantly positive for environmental governance. Second, from the perspective of sub-regions, effectiveness shows spatial heterogeneity across Eastern, Central, and Western China. Third, in Chinese-style AE, the public's environmental appeal has not received sufficient attention, and people's participation in environmental governance is limited. Finally, this study discusses ways to improve Chinese-style AE in China's institutional context and political system.

Script-Independent Text Segmentation from Document Images

Document image analysis finds broad application in the digital world for the purpose of information retrieval. This includes optical character recognition (OCR), indexing of digital libraries, web image processing, etc. One of the important steps in this field is text segmentation. This segmentation becomes complicated for the documents containing text of uneven spacing and characters of varying font sizes. In this paper, script-independent text-line segmentation and word segmentation algorithms are presented. Fast marching method is used for text-line segmentation, whereas wavelet transform with connected components (CCs) labeling is used for word segmentation. Fast marching method is used as a region growing process that detects potential text-lines. For word segmentation, energy map is calculated using wavelet transform to create text-blocks. Both the proposed algorithms are evaluated on different databases containing documents of different scripts, where highest text-line and word segmentation accuracies of 98.9% and 99.1%, respectively, are obtained.

A hybrid Chinese word segmentation model for quality management-related texts based on transfer learning.

Text information mining is a key step to data-driven automatic/semi-automatic quality management (QM). For Chinese texts, a word segmentation algorithm is necessary for pre-processing since there are no explicit marks to define word boundaries. Because of intrinsic characteristics of QM-related texts, word segmentation algorithms for normal Chinese texts cannot be directly applied. Hence, based on the analysis of QM-related texts, we summarized six features, and proposed a hybrid Chinese word segmentation model by means of integrating transfer learning (TL), bidirectional long-short term memory (Bi-LSTM), multi-head attention (MA), and conditional random field (CRF) to construct the mTL-Bi-LSTM-MA-CRF model, considering insufficient samples of QM-related texts and excessive cutting of idioms. The mTL-Bi-LSTM-MA-CRF model is composed of two steps. Firstly, based on a word embedding space, the Bi-LSTM is introduced for context information learning, and the MA mechanism is selected to allocate attention among subspaces, and then the CRF is used to learn label sequence constraints. Secondly, a modified TL method is put forward for text feature extraction, adaptive layer weights learning, and loss function correction for selective learning. Experimental results show that the proposed model can achieve good word segmentation results with only a relatively small set of samples.

KOHTD: Kazakh offline handwritten text dataset

Despite the transition to digital information exchange, many documents, such as invoices, taxes, memos and questionnaires, historical data, and answers to exam questions, still require handwritten inputs. In this regard, there is a need to implement Handwritten Text Recognition (HTR) which is an automatic way to decrypt records using a computer. Handwriting recognition is challenging because of the virtually infinite number of ways a person can write the same message. For this proposal we introduce Kazakh handwritten text recognition research, a comprehensive dataset of Kazakh handwritten texts is necessary. This is particularly true given the lack of a dataset for handwritten Kazakh text. In this paper, we proposed our extensive Kazakh offline Handwritten Text dataset (KOHTD), which has 3000 handwritten exam papers and more than 140335 segmented images and there are approximately 922010 symbols. It can serve researchers in the field of handwriting recognition tasks by using deep and machine learning. We used a variety of popular text recognition methods for word and line recognition in our studies, including CTC-based and attention-based methods. In this paper, we have implemented state-of-the-art deep learning-based methods for Handwriting recognition for KOHTD dataset to create several strong baselines. The findings demonstrate KOHTD’s diversity. Also, we proposed a Genetic Algorithm (GA) for line and word segmentation based on random enumeration of a parameter. The dataset and GA code are available at https://github.com/abdoelsayed2016/KOHTD.

Deep Neural Networks Algorithm for Vietnamese Word Segmentation

Traditional Vietnamese word segmentation methods do not perform well in the face of Vietnamese ambiguity, in response to the enormous challenge posed by the scarcity of the Vietnamese corpus to language processing. We first investigated the most advanced deep neural network method. According to the ambiguity problem of Vietnamese word segmentation, we then proposed a Vietnamese word segmentation processing technology based on an improved long short-term memory neural network (LSTM), which is made up of an LSTM encoding and a CNN feature extraction portion. The previous important information is kept in the memory unit; the word segmentation processing task is refined into a classification problem and a sequence labeling problem, which can gain the useful features of the word segmentation character and word level automatically. The limitation of the local context window size is avoided, and the word segmentation processing task is refined into a classification problem and a sequence labeling problem. Finally, validated by a homemade Vietnamese news website crawler dataset, the experimental results show that, compared with the single LSTM, single CNN methods, and traditional methods, the performance improvement of our proposed method is more obvious. In the Vietnamese word separation task, the accuracy reaches 96.6%, the recall reaches 95.2%, and the F1 value reaches 96.3%, which is significantly better than the traditional methods CNN and LSTM.

Does morphological complexity affect word segmentation? Evidence from computational modeling

How can infants detect where words or morphemes start and end in the continuous stream of speech? Previous computational studies have investigated this question mainly for English, where morpheme and word boundaries are often isomorphic. Yet in many languages, words are often multimorphemic, such that word and morpheme boundaries do not align. Our study employed corpora of two languages that differ in the complexity of inflectional morphology, Chintang (Sino-Tibetan) and Japanese (in Experiment 1), as well as corpora of artificial languages ranging in morphological complexity, as measured by the ratio and distribution of morphemes per word (in Experiments 2 and 3). We used two baselines and three conceptually diverse word segmentation algorithms, two of which rely purely on sublexical information using distributional cues, and one that builds a lexicon. The algorithms' performance was evaluated on both word- and morpheme-level representations of the corpora. Segmentation results were better for the morphologically simpler languages than for the morphologically more complex languages, in line with the hypothesis that languages with greater inflectional complexity could be more difficult to segment into words. We further show that the effect of morphological complexity is relatively small, compared to that of algorithm and evaluation level. We therefore recommend that infant researchers look for signatures of the different segmentation algorithms and strategies, before looking for differences in infant segmentation landmarks across languages varying in complexity.

Ancient Tibetan Word Segmentation: Dataset and Methods

Tibetan ancient literature is an important literature material for the study of ancient Tibetan culture, history, and the development of Sino Tibetan language family. However, the lack of work on ancient Tibetan word segmentation tools seriously restricts the research of ancient Tibetan literature. In view of this situation, this paper first utilizes ancient Tibetan interlaced contrast tagging data to extract the ancient Tibetan word segmentation dataset. Based on this dataset, we conduct numerous experiments for the task of ancient Tibetan word segmentation. Experimental results show that BiLSTM + CRF word segmentation algorithm can achieve the best performance, and the performance of ancient Tibetan word segmentation can be further improved through model ensemble. And the results show that the unknown words, insufficient training data and word ambiguity restrict the performance of ancient Tibetan word segmentation.

HMM-BiMM: Hidden Markov Model-based word segmentation via improved Bi-directional Maximal Matching algorithm

Combining with the Hidden Markov Model and Bi-directional Maximal Matching algorithm, a new word segmentation algorithm, HMM-BiMM, is presented. In terms of the sub-dictionary matching, it can implement a fast word segmentation. After segmenting the text by the Bi-directional Maximal Matching (BiMM), the remaining text connected by the remaining single words will be segmented again by the strategy of the Hidden Markov Model (HMM). By the HMM, this algorithm can realize the dictionary dynamic update by the new segmentation words and improve the segmentation accuracy accordingly. Compared with five representative algorithms in the real-world clinical text (symptom), we show that the HMM-BiMM algorithm achieves the highest efficiency and accuracy for symptom text segmentation. In detail, this algorithm has around 3% in precision and 70% in running time improved to the BiMM.

Research on Intelligent Search Framework Technology Oriented to Monitoring of Power Grid Dispatching Equipment

The development of power grid dispatching and monitoring services puts forward new requirements for the digitization and intelligence of dispatching systems. This paper combined grid control data and business characteristics to study the intelligent search framework for grid dispatch monitoring, as well as key technologies such as thesaurus modeling, power feature word segmentation algorithm, multi-factor correlation ranking algorithm, and power knowledge graph association modeling, and developed a smart search application for the power grid that regulates and controls the cloud, to implement search for power grid equipment models, equipment operations, monitoring events, services, etc. It can carry out comprehensive, accurate, fast and intelligent retrieval of the monitoring operation data of mass dispatching equipment, which provides strong support for the management and decision-making of dispatching equipment monitoring business.

Chinese Named Entity Recognition Based on Character-Word Vector Fusion

Due to the lack of explicit markers in Chinese text to define the boundaries of words, it is often more difficult to identify named entities in Chinese than in English. At present, the pretreatment of the character or word vector models is adopted in the training of the Chinese named entity recognition model. Aimed at the problems that taking character vector as an input of the neural network cannot use the words’ semantic meanings and give up the words’ explicit boundary information, and taking the word vector as an input of the neural network relies on the accuracy of the segmentation algorithms, a Chinese named entity recognition model based on character word vector fusion CWVF-BiLSTM-CRF (Character Word Vector Fusion-Bidirectional Long-Short Term Memory Networks-Conditional Random Field) is proposed in this paper. First, the Word2Vec is used to obtain the corresponding dictionaries of character-character vector and word-word vector. Second, the character-word vector is integrated as the input unit of the BiLSTM (Bidirectional Long-Short Term Memory) network, and then, the problem of an unreasonable tag sequence is solved using the CRF (conditional random field). By using the presented model, the dependence on the accuracy of the word segmentation algorithm is reduced, and the words’ semantic characteristics are effectively applied. The experimental results show that the model based on character-word vector fusion improves the recognition effect of the Chinese named entity.

MiNgMatch—A Fast N-gram Model for Word Segmentation of the Ainu Language

Word segmentation is an essential task in automatic language processing for languages where there are no explicit word boundary markers, or where space-delimited orthographic words are too coarse-grained. In this paper we introduce the MiNgMatch Segmenter—a fast word segmentation algorithm, which reduces the problem of identifying word boundaries to finding the shortest sequence of lexical n-grams matching the input text. In order to validate our method in a low-resource scenario involving extremely sparse data, we tested it with a small corpus of text in the critically endangered language of the Ainu people living in northern parts of Japan. Furthermore, we performed a series of experiments comparing our algorithm with systems utilizing state-of-the-art lexical n-gram-based language modelling techniques (namely, Stupid Backoff model and a model with modified Kneser-Ney smoothing), as well as a neural model performing word segmentation as character sequence labelling. The experimental results we obtained demonstrate the high performance of our algorithm, comparable with the other best-performing models. Given its low computational cost and competitive results, we believe that the proposed approach could be extended to other languages, and possibly also to other Natural Language Processing tasks, such as speech recognition.

Cultural communication in double-layer coupling social network based on association rules in big data

In the big data set around the world, cultural communication in a single-layer social network cannot handle data in a two-tier social network. Combined with the characteristics of cultural communication, a two-layer coupled social network model for predicting cultural information dissemination is proposed. According to the law of big data set in the process of cultural communication, the ethical review and theoretical construction of cultural network communication was carried out. Combining the online two-layer coupling network and the CRF word segmentation algorithm, the known link relationship is obtained, and the feature attributes of the network are extracted. This paper mainly completed the construction of online social network link prediction. Starting from the topological structure of two-layer coupled social network, based on the association rules to analyze the data generated by information dissemination between two-layer coupled network nodes, a two-layer coupled network information propagation law model is proposed. The prediction and construction of the cultural information communication link in the online social network is completed, the process of data opening and sharing is accelerated, and the privacy and copyright issues in the process of cultural communication are solved. The influence of immune node and node self-healing time on the cultural information dissemination process of two-layer coupled social network is discussed. By comparing the information dissemination in social networks and the information dissemination in two-layer coupled social networks, the control of certain lyric cultures can be effectively prevented. The experimental results show that it has reference value.

An efficient, font independent word and character segmentation algorithm for printed Arabic text

Characters segmentation is a necessity and the most critical stage in Arabic OCR system. It has attracted the interest of a wide range of researchers. However, the nature of the Arabic cursive script poses extra challenges that need further investigation. Therefore, having a reliable and efficient Arabic OCR system that is independent of font variations is highly required. In this paper, an indirect, font-in dependent word and character segmentation algorithm for printed Arabic text investigated. The proposed algorithm takes a binary line image as an input and produces a set of binary images consisting of one character or ligature as an output. The segmentation performed at two levels: a word segmentation performed in the first level, by employing a vertical projection at the input line image along with using Interquartile Range (IQR) method to differentiate between word gaps and within word gaps. A projection profile method used as a second level of segmentation along with a set of statistical and topological features, which are font-independent, to identify the correct segmentation points from all potential points. The APTI dataset used to test the proposed algorithm with a variety of font type, size, and style. The algorithm experimented on 1800 lines (approximately 24,816 words) with an average accuracy of 97.7% for words segmentation and 97.51% for characters segmentation.

Hot Topic Detection Based on a Refined TF-IDF Algorithm

In this paper, we propose a refined term frequency inversed document frequency (TF-IDF) algorithm called TA TF-IDF to find hot terms, based on time distribution information and user attention. We also put forward a method to generate new terms and combined terms, which are split by the Chinese word segmentation algorithm. Then, we extract hot news according to the hot terms, grouping them into K-means clusters so as to realize the detection of hot topics in news. The experimental results indicated that our method based on the refined TF-IDF algorithm can find hot topics effectively.

C-RNN: A Fine-Grained Language Model for Image Captioning

Captioning methods from predecessors that based on the conventional deep Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) architecture follow translation system using word-level modelling. But an optimal word segmentation algorithm is essential for segmenting sentence into words in word-level modelling, which is a very difficult task. In this paper, we built a character-level RNN (c-RNN) that directly modeled on captions with characterization where descriptive sentence is composed in a flow of characters. The c-RNN performs language task in finer level and naturally avoids the word segmentation issue. Our c-RNN empowered the language model to dynamically reason about word spelling as well as grammatical rules which results in expressive and elaborate sentence. We optimized parameters of neural nets by maximizing the probabilities of correctly generated characterized sentences. Quantitative and qualitative experiments on the most popular datasets MSCOCO and Flickr30k showed that our c-RNN could describe images with a considerably faster speed and satisfactory quality.

Analysis of Consumer Appraisal of China’s Rural Specialty E-Commerce under Data Mining Method

In the research, the crawler technology was used to evaluate the Taobao silkie, and product evaluation data is the case object. After word segmentation, naive Bayesian, K-means, and TF-IDF related algorithms were used to complete word frequency statistics, sentiment analysis, and cluster analysis. It was found that the quality of rural specialty products was more recognized by e-commerce customers, but its overall emotional score was slightly lower than that of non-origin products. After summing up the drawbacks of the former, targeted countermeasures and suggestions were proposed.

Алгоритм сегментації слів на основі пошуку найкоротшого шляху в графі

The features of word segmentation algorithms from such texts are considered. There are two main models, namely, statistical one and the one using a dictionary. For models with a dictionary, a variant of the maximal matching algorithm is noted for which there are modifications such as Forward Maximal Matching (FMM) and Backward Maximal Matching (BMM) to be depending on the direction of text processing. The second option for models with a dictionary is an algorithm that finds segmentation with a minimum number of words. A new algorithm for words segmenting based on a modified wave algorithm has been presented. The algorithm takes into account the features of the input data and is built in such a way that the necessary calculations are performed in a single pass. This reduces its computational complexity. A description of the word segmentation algorithm is given. An example is shown of splitting an input string in English into words, representing it in the form of a graph and finding the shortest path.To assess the quality of segmentation, the EDWS (Edit Distance of the Word Separator) method is presented. A special tool was used to assess the segmentation of Chinese words with a test corpora based on news texts. Evaluations of the quality of segmentation of words for the proposed algorithm (based on the search for the shortest path) and a number of other known segmentators are obtained. An example of segmentation of a news text in Russian is given. The possibilities of using the developed algorithm in the problems of information search in national resources of the Internet are shown. The implementation of the word segmentation algorithm is used when creating a generalized domain model based on monitoring of the Chinese Internet segment resources.The increase in the number of information resources of the Chinese Internet segment makes it necessary to create of global information retrieval systems. For search indexes of such systems, fast, accurate and complete segmentation of words from texts is necessary. The obtained estimates of segmentation quality using the proposed algorithm for the formation of the search system index indicate the possibility of its use for information resources of the Chinese Internet-segment.

Analysis on Applicability of Common Chinese Word Segmentation Software in Literature Study of Traditional Chinese Medicine Text

This study was aimed to evaluate the applicability of common Chinese word segmentation software used in the literature study of traditional Chinese medicine (TCM) text, in order to put forward ideas on developing specialized TCM text word segmentation software. By means of installing and operating Chinese word segmentation software, the word segmentation experiment was conducted on TCM text samples. Aspects, such as Chinese word segmentation accuracy, speed, maneuverability, reliability, extendibility, portability and other characteristics, were compared among different Chinese word segmentation software. The results showed that there were differences on the accuracy, speed, maneuverability, reliability, extendibility, portability among different Chinese word segmentation software. It was difficult to achieve best performance on different aspects by single software. Through the comparison of different Chinese word segmentation software, the Pan-Gu Segment software showed the best performance on accuracy, with good maneuverability, and high word segmentation efficiency, which was the most suitable for word segmentation in TCM text. It was concluded that developing specialized TCM word segmentation software may be the best solution to meet the requirement of word segmentation in TCM literature study. Basic studies should be strengthened from aspects, such as the construction of standard TCM corpus, the completion of TCM dictionary base, the introduction, optimization and innovation of word segmentation algorithm, as well as the development of word segmentation software for TCM text.