Unsupervised Word Segmentation Research Articles

A traffic anomaly detection approach based on unsupervised learning for industrial cyber–physical system

In Industrial Cyber–Physical Systems (ICPSs), the attacker can intrude into the cyber system through many penetration tools and attack the physical system. Payload-based traffic anomaly detection is a popular technique against these attacks. Due to the imbalanced distribution of normal and attack samples in ICPS, existing payload-based detection methods are mostly implemented based on unsupervised learning, typically comprising a word segmentation model and an unsupervised classifier. However, existing methods may disrupt semantic correlations and face challenges in extracting complex payload dependence relationships. To address these issues, this paper proposes a traffic anomaly detection approach, which consists of a data preprocessing model, an unsupervised word segmentation model, and an unsupervised classification model based on autoencoder. The unsupervised word segmentation model utilizes Long Short-Term Memory (LSTM) to calculate the probability of each word segmentation combination, effectively addressing the issue of inaccurate segmentation results in existing payload segmentation models. The unsupervised classification model, which combines 1D-Convolutional Neural Network (1D-CNN) and Bidirectional Encoder Representation from Transformers (BERT), addresses the challenge of extracting complex payload dependence relationships in existing classification models. The proposed detection approach is evaluated using a Cyber–Physical Attack Dataset (CPAD). Compared with the state-of-the-art detection approaches, the proposed approach has shown a significant improvement in Precision, with an increase of 18.83%. Additionally, the Recall has also been substantially enhanced, with a gain of 22.3%. Overall, the F1 has demonstrated a comprehensive improvement of 20.60%.

BERTCWS: unsupervised multi-granular Chinese word segmentation based on a BERT method for the geoscience domain

ABSTRACT Unlike alphabet-based languages such as English, the Chinese language has no specifying word boundaries. Segmentation, particularly for the Chinese language, is a fundamental step towards Chinese text processing, information retrieval, and knowledge discovery. In the geoscience domain, most existing Chinese word segmentation tools/models require a prespecified dictionary and a large amount of relevant training corpus, and the segmentation accuracies drop significantly when processing out-domain situations using these same methods. To address this issue, a purely unsupervised and generic two-stage architecture (named BERTCWS) for domain-specific Chinese word segmentation is proposed. We first design an incidence matrix termed the ‘character combination tightness’ to calculate the closeness between characters. Then, BERTCWS recognizes geoscience terms based on a Bidirectional Encoder Representations from Transformers(BERT)-based segmenter, and multi-granular segmentation is generated by setting different thresholds. Finally, the discriminator is constructed to validate the correctness of the segmented words. Our numerical study demonstrates that BERTCWS can identify both general-domain terms and geoscience-domain terms. Additionally, multi-granular segmentation could be applied to offer a set of potential geoscience terms of various lengths.

Unsupervised Word Segmentation with Bi-directional Neural Language Model

We propose an unsupervised word segmentation model, in which for each unlabelled sentence sample, the learning objective is to maximize the generation probability of the sentence given its all possible segmentations. Such a generation probability can be factorized into the likelihood of each possible segment given the context in a recursive way. To capture both the long- and short-term dependencies, we propose to use a bi-directional neural language model to better extract the features of the segment’s context. Two decoding algorithms were also developed to combine the context features from both directions to generate the final segmentation at the inference time, which helps to reconcile word-boundary ambiguities. Experimental results show that our context-sensitive unsupervised segmentation model achieved state-of-the-art at different evaluation settings on various datasets for Chinese, and the comparable result for Thai.

An Unsupervised and Robust Line and Word Segmentation Method for Handwritten and Degraded Printed Document

Segmentation of text lines and words in an unconstrained handwritten or a machine-printed degraded document is a challenging document analysis problem due to the heterogeneity in the document structure. Often there is un-even skew between the lines and also broken words in a document. In this article, the contribution lies in segmentation of a document page image into lines and words. We have proposed an unsupervised, robust, and simple statistical method to segment a document image that is either handwritten or machine-printed (degraded or otherwise). In our proposed method, the segmentation is treated as a two-class classification problem. The classification is done by considering the distribution of gap size (between lines and between words) in a binary page image. Our method is very simple and easy to implement. Other than the binarization of the input image, no pre-processing is necessary. There is no need of high computational resources. The proposed method is unsupervised in the sense that no annotated document page images are necessary. Thus, the issue of a training database does not arise. In fact, given a document page image, the parameters that are needed for segmentation of text lines and words are learned in an unsupervised manner. We have applied our proposed method on several popular publicly available handwritten and machine-printed datasets (ISIDDI, IAM-Hist, IAM, PBOK) of different Indian and other languages containing different fonts. Several experimental results are presented to show the effectiveness and robustness of our method. We have experimented on ICDAR-2013 handwriting segmentation contest dataset and our method outperforms the winning method. In addition to this, we have suggested a quantitative measure to compute the level of degradation of a document page image.

Culture under Complex Perspective: A Classification for Traditional Chinese Cultural Elements Based on NLP and Complex Networks

The cultural element is the minimum unit of a cultural system. The systematic categorizing, organizing, and retrieval of the traditional Chinese cultural elements are essential prerequisites for the realization of effective extracting and rational utilization, as well as the prerequisite for exploiting the contemporary value of the traditional Chinese culture. To build an objective, integrated, and reliable classification method and a system of traditional Chinese cultural elements, this study takes the text of Taiping Imperial Encyclopedia in Northern Song Dynasty as the primary data source. The unsupervised word segmentation methods are used to detect Out-of-Vocabulary (OOV), and then the segmentation results by the THULAC tool with and without custom dictionary are compared. The TF-IDF algorithm is applied to extract the keywords of cultural elements and the Ochiia coefficient is introduced to create complex networks of traditional Chinese cultural elements. After analyzing the topological characteristics of the network, the community detection algorithm is used to identify the topics of cultural elements. Finally, a “Means-Ends” two-dimensional orthogonal classification system is established to categorize the topics. The results showed that the degree distribution in the complex network of Chinese traditional cultural elements is a scale-free network with γ = 2.28. The network shows a structure of community and hierarchy features. The top 12 communities have taken up to 91.77% of the scale of the networks. Those 12 topics of the traditional Chinese cultural elements are circularly distributed in the orthogonal system of cultural elements’ categorization.

Word Segmentation for Classical Chinese Buddhist Literature

With the growth of digital humanities, information technologies take on more important roles in humanities research, including the study of religion. To analyze text for further processing, many text analysis tools treat a word as a unit. However, in Chinese, there are no word boundary markers. Word segmentation is required for processing Chinese texts. Although several word segmentation tools are available for modern Chinese, there is still no practical word segmentation tool for Classical Chinese, especially for Classical Chinese Buddhist literature. In this paper, we adopt unsupervised and supervised learning techniques to build Classical Chinese word segmentation approaches for processing Buddhist literature. Normalized variation of branching entropy (nVBE) is adopted for unsupervised word segmentation. Conditional random fields (CRF) are used to generate supervised models for Classical Chinese word segmentation. The performance of our word segmentation approach achieves an F-score of up to 0.9396. The experimental results show that our proposed method is effective for correctly segmenting most Classical Chinese sentences in Buddhist literature. Our word segmentation method can be a fundamental tool for further text analysis and processing research, such as word embedding, syntactic parsing, and semantic labeling.

Investigating alignment interpretability for low-resource NMT

The attention mechanism in Neural Machine Translation (NMT) models added flexibility to translation systems, and the possibility to visualize soft-alignments between source and target representations. While there is much debate about the relationship between attention and the yielded output for neural models (Jain and Wallace 2019; Serrano and Smith 2019; Wiegreffe and Pinter 2019; Vashishth et al. 2019), in this paper we propose a different assessment, investigating soft-alignment interpretability in low-resource scenarios. We experimented with different architectures (RNN (Bahdanau et al. 2015), 2D-CNN (Elbayad et al. 2018), and Transformer (Vaswani et al. 2017)), comparing them with regards to their ability to produce directly exploitable alignments. For evaluating exploitability, we replicated the Unsupervised Word Segmentation (UWS) task from Godard et al. (2018). There, source words are translated into unsegmented phone sequences. Posterior to training, the resulting soft-alignments are used for producing segmentation over the target side. Our results showed that a RNN-based NMT model produced the most exploitable alignments in this scenario. We then investigated methods for increasing its UWS scores by comparing the following methodologies: monolingual pre-training, input representation augmentation (hybrid model), and explicit word length optimization during training. We reached the best results by using the hybrid model, which uses an intermediate monolingual-rooted segmentation from a non-parametric Bayesian model (Goldwater 2007) to enrich the input representation before training.

Unsupervised multi-granular Chinese word segmentation and term discovery via graph partition.

This study aims at realizing unsupervised term discovery in Chinese electronic health records (EHRs) by using the word segmentation technique. The existing supervised algorithms do not perform satisfactorily in the case of EHRs, as annotated medical data are scarce. We propose an unsupervised segmentation method (GTS) based on the graph partition principle, whose multi-granular segmentation capability can help realize efficient term discovery. A sentence is converted to an undirected graph, with the edge weights based on n-gram statistics, and ratio cut is used to split the sentence into words. The graph partition is solved efficiently via dynamic programming, and multi-granularity is realized by setting different partition numbers. A BERT-based discriminator is trained using generated samples to verify the correctness of the word boundaries. The words that are not recorded in existing dictionaries are retained as potential medical terms. We compared the GTS approach with mature segmentation systems for both word segmentation and term discovery. MD students manually segmented Chinese EHRs at fine and coarse granularity levels and reviewed the term discovery results. The proposed unsupervised method outperformed all the competing algorithms in the word segmentation task. In term discovery, GTS outperformed the best baseline by 17 percentage points (a 47% relative percentage of increment) on F1-score. In the absence of annotated training data, the graph partition technique can effectively use the corpus statistics and even expert knowledge to realize unsupervised word segmentation of EHRs. Multi-granular segmentation can be used to provide potential medical terms of various lengths with high accuracy.

Multilingual and unsupervised subword modeling for zero-resource languages

Subword modeling for zero-resource languages aims to learn low-level representations of speech audio without using transcriptions or other resources from the target language (such as text corpora or pronunciation dictionaries). A good representation should capture phonetic content and abstract away from other types of variability, such as speaker differences and channel noise. Previous work in this area has primarily focused unsupervised learning from target language data only, and has been evaluated only intrinsically. Here we directly compare multiple methods, including some that use only target language speech data and some that use transcribed speech from other (non-target) languages, and we evaluate using two intrinsic measures as well as on a downstream unsupervised word segmentation and clustering task. We find that combining two existing target-language-only methods yields better features than either method alone. Nevertheless, even better results are obtained by extracting target language bottleneck features using a model trained on other languages. Cross-lingual training using just one other language is enough to provide this benefit, but multilingual training helps even more. In addition to these results, which hold across both intrinsic measures and the extrinsic task, we discuss the qualitative differences between the different types of learned features.

Nonparametric Bayesian Semi-supervised Word Segmentation

This paper presents a novel hybrid generative/discriminative model of word segmentation based on nonparametric Bayesian methods. Unlike ordinary discriminative word segmentation which relies only on labeled data, our semi-supervised model also leverages a huge amounts of unlabeled text to automatically learn new “words”, and further constrains them by using a labeled data to segment non-standard texts such as those found in social networking services. Specifically, our hybrid model combines a discriminative classifier (CRF; Lafferty et al. (2001) and unsupervised word segmentation (NPYLM; Mochihashi et al. (2009)), with a transparent exchange of information between these two model structures within the semi-supervised framework (JESS-CM; Suzuki and Isozaki (2008)). We confirmed that it can appropriately segment non-standard texts like those in Twitter and Weibo and has nearly state-of-the-art accuracy on standard datasets in Japanese, Chinese, and Thai.

Unsupervised spatial lexical acquisition by updating a language model with place clues

Abstract This paper describes how to achieve highly accurate unsupervised spatial lexical acquisition from speech-recognition results including phoneme recognition errors. In most research into lexical acquisition, the robot has no pre-existing lexical knowledge. The robot acquires sequences of some phonemes as words from continuous speech signals. In a previous study, we proposed a nonparametric Bayesian spatial concept acquisition method (SpCoA) that integrates the robot’s position and words obtained by unsupervised word segmentation from uncertain syllable recognition results. However, SpCoA has a very critical problem to be solved in lexical acquisition; the boundaries of word segmentation are incorrect in many cases because of many phoneme recognition errors. Therefore, we propose an unsupervised machine learning method (SpCoA++) for the robust lexical acquisition of novel words relating to places visited by the robot. The proposed SpCoA++ method performs an iterative estimation of learning spatial concepts and updating a language model using place information. SpCoA++ can select a candidate including many words that better represent places from multiple word-segmentation results by maximizing the mutual information between segmented words and spatial concepts. The experimental results demonstrate a significant improvement of the phoneme accuracy rate of learned words relating to place in the proposed method by word-segmentation results based on place information, in comparison to the conventional methods. We indicate that the proposed method enables the robot to acquire words from speech signals more accurately, and improves the estimation accuracy of the spatial concepts.

Online Algorithm for Robots to Learn Object Concepts and Language Model

Humans form concept of objects by classifying them into categories, and acquire language by simultaneously interacting with others. Thus, the meaning of a word can be learned by connecting a recognized word to its corresponding concept. We consider this ability important for robots to flexibly develop knowledge of language and concepts. In this paper, we propose an online algorithm for robots to acquire knowledge of natural language and learn object concepts. A robot learns the language model from word sequences, which are obtained by the segmentation of phoneme sequences provided by a user, by using unsupervised word segmentation each time it is provided with a new object. Moreover, the robot acquires object concepts using these word sequences as well as multimodal information obtained by observing objects. The crucial aspect of our model is the interdependence of words and concepts: there is a high probability that the same words will be uttered to describe objects in the same category. By taking this relationship into account, our proposed method enables robots to acquire a more accurate language model and object concepts online. Experimental results verify this.

Sentiment detection in micro-blogs using unsupervised chunk extraction

In this paper, we present a proposed system designed for sentiment detection for micro-blog data in Chinese. Our system surprisingly benefits from the lack of word boundary in Chinese writing system and shifts the focus directly to larger and more relevant chunks. We use an unsupervised Chinese word segmentation system and binomial test to extract specific and endogenous lexicon chunks from the training corpus. We combine the lexicon chunks with other external resources to train a maximum entropy model for document classification. With this method, we obtained an averaged F1 score of 87.2 which outperforms the state-of-the-art approach based on the released data in the second SocialNLP shared task.

Spatial Concept Acquisition for a Mobile Robot that Integrates Self-Localization and Unsupervised Word Discovery from Spoken Sentences

In this paper, we propose a novel unsupervised learning method for the lexical acquisition of words related to places visited by robots, from human continuous speech signals. We address the problem of learning novel words by a robot that has no prior knowledge of these words except for a primitive acoustic model. Further, we propose a method that allows a robot to effectively use the learned words and their meanings for self-localization tasks. The proposed method is nonparametric Bayesian spatial concept acquisition method (SpCoA) that integrates the generative model for self-localization and the unsupervised word segmentation in uttered sentences via latent variables related to the spatial concept. We implemented the proposed method SpCoA on SIGVerse, which is a simulation environment, and TurtleBot2, which is a mobile robot in a real environment. Further, we conducted experiments for evaluating the performance of SpCoA. The experimental results showed that SpCoA enabled the robot to acquire the names of places from speech sentences. They also revealed that the robot could effectively utilize the acquired spatial concepts and reduce the uncertainty in self-localization.

Detecting “protein words” through unsupervised word segmentation

Unsupervised word segmentation methods were applied to analyze protein sequences. Protein sequences, such as “MTMDKSELVQKA…,” were used as input to these methods. Segmented protein word sequences, such as “MTM DKSE LVQKA,” were then obtained. We compared the protein words derived via unsupervised segmentation and protein secondary structure segmentation. An interesting finding is that unsupervised word segmentation is more efficient than secondary structure segmentation in expressing information. Our experiment also suggests the presence of several “protein ruins” in current non-coding regions.

Online Adaptor Grammars with Hybrid Inference

Adaptor grammars are a flexible, powerful formalism for defining nonparametric, unsupervised models of grammar productions. This flexibility comes at the cost of expensive inference. We address the difficulty of inference through an online algorithm which uses a hybrid of Markov chain Monte Carlo and variational inference. We show that this inference strategy improves scalability without sacrificing performance on unsupervised word segmentation and topic modeling tasks.

A Lexicon-Corpus-based Unsupervised Chinese Word Segmentation Approach

Abstract This paper presents a Lexicon-Corpus-based Unsupervised (LCU) Chinese word segmentation approach to improve the Chinese word segmentation result. Specifically, it combines advantages of lexicon-based approach and Corpus-based approach to identify out-of-vocabulary (OOV) words and guarantee segmentation consistency of the actual words in texts as well. In addition, a Forward Maximum Fixed-count Segmentation (FMFS) algorithm is developed to identify phrases in texts at first. Detailed rules and experiment results of LCU are presented, too. Compared with lexicon-based approach or corpus-based approach, LCU approach makes a great improvement in Chinese word segmentation, especially for identifying n-char words. And also, two evaluation indexes are proposed to describe the effectiveness in extracting phrases, one is segmentation rate (S), and the other is segmentation consistency degree (D).

An Efficient Algorithm for Unsupervised Word Segmentation with Branching Entropy and MDL

This paper proposes a fast and simple unsupervised word segmentation algorithm that utilizes the local predictability of adjacent character sequences, while searching for a least-effort representation of the data. The model uses branching entropy as a means of constraining the hypothesis space, in order to efficiently obtain a solution that minimizes the length of a two-part MDL code. An evaluation with corpora in Japanese, Thai, English, and the CHILDES corpus for research in language development reveals that the algorithm achieves an accuracy, comparable to that of the state-of-the-art methods in unsupervised word segmentation, in a significantly reduced computational time.

An Efficient Algorithm for Unsupervised Word Segmentation with Branching Entropy and MDL

An Efficient Algorithm for Unsupervised Word Segmentation with Branching Entropy and MDL

A Simple and Effective Unsupervised Word Segmentation Approach

In this paper, we propose a new unsupervised approach for word segmentation. The core idea of our approach is a novel word induction criterion called WordRank, which estimates the goodness of word hypotheses (character or phoneme sequences). We devise a method to derive exterior word boundary information from the link structures of adjacent word hypotheses and incorporate interior word boundary information to complete the model. In light of WordRank, word segmentation can be modeled as an optimization problem. A Viterbi-styled algorithm is developed for the search of the optimal segmentation. Extensive experiments conducted on phonetic transcripts as well as standard Chinese and Japanese data sets demonstrate the effectiveness of our approach. On the standard Brent version of Bernstein-Ratner corpora, our approach outperforms the state-of-the-art Bayesian models by more than 3%. Plus, our approach is simpler and more efficient than the Bayesian methods. Consequently, our approach is more suitable for real-world applications.