Performance Of Word Segmentation Research Articles

A Levenshtein distance-based method for word segmentation in corpus augmentation of geoscience texts

ABSTRACT For geoscience text, rich domain corpora have become the basis of improving the model performance in word segmentation. However, the lack of domain-specific corpus with annotation labelled has become a major obstacle to professional information mining in geoscience fields. In this paper, we propose a corpus augmentation method based on Levenshtein distance. According to the technique, a geoscience dictionary of 20,137 words was collected and constructed by crawling the keywords from published papers in China National Knowledge Infrastructure (CNKI). The dictionary was further used as the main source of synonyms to enrich the geoscience corpus according to the Levenshtein distance between words. Finally, a Chinese word segmentation model combining the BERT, Bi-gated recurrent neural network (Bi-GRU), and conditional random fields (CRF) was implemented. Geoscience corpus composed of complex long specific vocabularies has been selected to test the proposed word segmentation framework. CNN-LSTM, Bi-LSTM-CRF, and Bi-GRU-CRF models were all selected to evaluate the effects of Levenshtein data augmentation technique. Experiments results prove that the proposed methods achieve a significant performance improvement of more than 10%. It has great potential for natural languages processing tasks like named entity recognition and relation extraction.

LATTE: Lattice ATTentive Encoding for Character-based Word Segmentation

A character sequence comprises at least one or more segmentation alternatives. This can be considered segmentation ambiguity and may weaken segmentation performance in word segmentation. Proper handling of such ambiguity lessens ambiguous decisions on word boundaries. Previous works have achieved remarkable segmentation performance and alleviated the ambiguity problem by incorporating the lattice, owing to its ability to capture segmentation alternatives, along with graph-based and pre-trained models. However, multiple granularity information, including character and word, in a lattice that encodes with such models may not be attentively exploited. To strengthen multi-granularity representations in a lattice, we propose the Lattice ATTentive Encoding (LATTE) method for character-based word segmentation. Our model employs the lattice structure to handle segmentation alternatives and utilizes graph neural networks along with an attention mechanism to attentively extract multi-granularity representation from the lattice for complementing character representations. Our experimental results demonstrated improvements in segmentation performance on the BCCWJ, CTB6, and BEST2010 datasets in three languages, particularly Japanese, Chinese, and Thai.

The role of mother-infant emotional synchrony in speech processing in 9-month-old infants

Rhythmicity characterizes both interpersonal synchrony and spoken language. Emotions and language are forms of interpersonal communication, which interact with each other throughout development. We investigated whether and how emotional synchrony between mothers and their 9-month-old infants relates to infants’ word segmentation as an early marker of language development. Twenty-six 9-month-old infants and their German-speaking mothers took part in the study. To measure emotional synchrony, we coded positive, neutral and negative emotional expressions of the mothers and their infants during a free play session. We then calculated the degree to which the mothers’ and their infants’ matching emotional expressions followed a predictable pattern. To measure word segmentation, we familiarized infants with auditory text passages and tested how long they looked at the screen while listening to familiar versus novel words. We found that higher levels of predictability (i.e. low entropy) during mother-infant interaction is associated with infants’ word segmentation performance. These findings suggest that individual differences in word segmentation relate to the complexity and predictability of emotional expressions during mother-infant interactions.

Investigating word segmentation of Chinese second language learners

Language-specific features necessitate certain processes and skills in reading. The visually unmarked between-word boundaries in written Chinese render it critical that readers be able to segment words in the continuous texts. It may pose challenges for second language (L2) readers whose first language (L1) is word-spaced. In light of the lack of understanding of Chinese L2 readers’ word segmentation, the present study investigated 100 L2 learners’ word segmentation performance, the relationships between word segmentation and reading fluency and comprehension, and the differences in the aforementioned relationships among learners with different context-free word recognition abilities. Results demonstrated that L2 learners generally conducted word segmentation well and word segmentation contributed to reading fluency and comprehension beyond context-free word recognition. Word segmentation was the major predictor of reading fluency and comprehension among weak wordlist readers, whereas context-free word recognition made a larger contribution when learners have achieved stronger context-free word recognition ability. The findings suggested the importance of developing word segmentation skills and establishing high quality word representations in Chinese L2 reading instruction.

Thai Word Segmentation Based on Sequence-to-Sequence Model

Thai as a low-resource language has a large word segmentation performance improvement space. In this paper, we investigate a sequence-to-sequence model for Thai word segmentation with two different recurrent neural networks, which could transform one input sequence into another output sequence. Furthermore, we evaluate datasets in four different fields compared then with other multiple word segmentation models, and the F1 value in the encyclopedia dataset reaches 97.15%. The results show that the proposed model has superior performance and is more effective, it is worth mentioning that the expected results can be achieved even with limited data resources.

Transliteration recognition of Tibetan person name based on Tibetan cultural knowledge

Institute of Computing Technology, Chinese Lexical Analysis System (ICTCLAS) is a common tool for Chinese word segmentation and named entity recognition. With this tool, the F1 value of person name recognition from Chinese texts in Tibetan culture is only 40%. We propose a method for transliteration recognition of Tibetan person name based on Tibetan cultural knowledge. Firstly, we leverage Tibetan cultural dictionary to improve the word segmentation performance of ICTCLAS. Then, special contextual features and naming rules of Tibetan person name in their culture are adopted to determine the boundary of Chinese transliteration of Tibetan person name. Finally, the transliteration candidates of Tibetan person name are filtered based on discrimination and reliability. Experiments on a 1.2 M Tibetan text in Tibetan culture show that the method can increase the F1 value of Chinese transliteration recognition of Tibetan person name from 40.08% to 87.92% in ICTCLAS.

Does visual speech information affect word segmentation?

We present an experiment in which we explored the extent to which visual speech information affects learners' ability to segment words from a fluent speech stream. Learners were presented with a set of sentences consisting of novel words, in which the only cues to the location of word boundaries were the transitional probabilities between syllables. They were exposed to this language through the auditory modality only, through the visual modality only (where the learners saw the speaker producing the sentences but did not hear anything), or through both the auditory and visual modalities. The learners were successful at segmenting words from the speech stream under all three training conditions. These data suggest that visual speech information has a positive effect on word segmentation performance, at least under some circumstances.

Speech segmentation by statistical learning depends on attention

We addressed the hypothesis that word segmentation based on statistical regularities occurs without the need of attention. Participants were presented with a stream of artificial speech in which the only cue to extract the words was the presence of statistical regularities between syllables. Half of the participants were asked to passively listen to the speech stream, while the other half were asked to perform a concurrent task. In Experiment 1, the concurrent task was performed on a separate auditory stream (noises), in Experiment 2 it was performed on a visual stream (pictures), and in Experiment 3 it was performed on pitch changes in the speech stream itself. Invariably, passive listening to the speech stream led to successful word extraction (as measured by a recognition test presented after the exposure phase), whereas diverted attention led to a dramatic impairment in word segmentation performance. These findings demonstrate that when attentional resources are depleted, word segmentation based on statistical regularities is seriously compromised.

Handwritten phrase recognition as applied to street name images

A phrase recognition method for recognition of street name images is presented in this paper. Some of the challenges posed by the problem are: (i) patron errors, (ii) non-standardized way of abbreviating names, and (iii) variable number of words in a street name image. A neural network has been designed to segment words in a phrase, using distance between components and style of writing. Experiments show perfect word segmentation performance of 85%. Substring matching is attempted only between the main body of a lexicon entry and the word segments of an image. Efforts to reduce computational complexity are successfully made by the sharing of character segmentation results between the segmentation and recognition phases. 83% phrase recognition accuracy was achieved on a test set.

World isolation and meaning in segmentation

The effect of meaning and isolation of words on the acquisition of segmentation of words in sentences was examined with an artificial miniature language composed of Mandarin Chinese quasi-elements. Forty undergraduate students participated in one of four groups: Word No Meaning, Word Meaning, Sentence No Meaning, or Sentence Meaning. In each group the subjects listened to the same set of words, either as isolated words or in sentences. Additionally, the meaning groups saw accompanying pictorial representations of isolated words or sentences. Pretest and posttest assessments indicated that isolation of words is a significant factor in word segmentation for an initial word set restricted in size when meaning is not provided. When meaning is provided, the isolation of words apparently does not influence word segmentation performance.