Word Segmentation Model Research Articles

On the segmentation of Chinese words during reading

Given that there are no spaces between words in Chinese, how words are segmented when reading is something of a mystery. Four Chinese characters, which either constituted one 4-character word or two 2-character words, were shown briefly to subjects. Subjects were quite accurate in reporting the 4-character word, but could usually only report the first 2-character word, demonstrating that word segmentation influences character recognition. The results suggest that even with these simple 4-character strings, there is an element of seriality in reading Chinese words: processing is initially focused at least to some extent on the first word. We also found that the processing of characters that are not consistent with the context is inhibited, suggesting inhibition from word representations to character representations. A simple model of Chinese word segmentation and word recognition is presented to account for the data.

Joint Chinese Word Segmentation and POS Tagging Using an Error-Driven Word-Character Hybrid Model

In this paper, we present a discriminative word-character hybrid model for joint Chinese word segmentation and POS tagging. Our word-character hybrid model offers high performance since it can handle both known and unknown words.We describe our strategies that yield good balance for learning the characteristics of known and unknown words and propose an error-driven policy that delivers such balance by acquiring examples of unknown words from particular errors in a training corpus. We describe an efficient framework for training our model based on the Margin Infused Relaxed Algorithm (MIRA), evaluate our approach on the Penn Chinese Treebank, and show that it achieves superior performance compared to the state-of-the-art approaches reported in the literature.

A role for backward transitional probabilities in word segmentation?

A number of studies have shown that people exploit transitional probabilities between successive syllables to segment a stream of artificial continuous speech into words. It is often assumed that what is actually exploited are the forward transitional probabilities (given XY, the probability that X will be followed by Y), even though the backward transitional probabilities (the probability that Y has been preceded by X) were equally informative about word structure in the languages involved in those studies. In two experiments, we showed that participants were able to learn the words from an artificial speech stream when the only available cues were the backward transitional probabilities. Learning is as good under those conditions as when the only available cues are the forward transitional probabilities. Implications for some current models of word segmentation, particularly the simple recurrent networks and PARSER models, are discussed.

Boundary alignment enables 11-month-olds to segment vowel initial words from speech

Past research has indicated that English-learning infants begin segmenting words from speech by 7-5 months of age (Jusczyk & Aslin, 1995). More recent work has demonstrated, however, that 7-5-month-olds' segmentation abilities are severely limited. For example, the ability to segment vowel-initial words from speech reportedly does not appear until 13.5 to 16 months of age (Mattys & Jusczyk, 2001; Nazzi, Dilley, Jusczyk, Shattuck-Hufnagel & Jusczyk, 2005). In this paper, we report on three experiments using the Headturn Preference procedure that investigate both phonetic and phonological factors influencing 11-month-olds' segmentation of vowel-initial words from speech. We replicate earlier findings suggesting that infants have difficulty segmenting vowel-initial words from speech. In addition we extend these findings by demonstrating that under certain conditions, infants are capable of segmenting vowel-initial words from speech at a much younger age than earlier studies have reported. Our findings suggest that infants' ability to segment vowel-initial words from speech is tightly constrained by acoustic-phonetic factors such as pitch movement at the onset of vowel-initial words and segmental strengthening. These experiments underscore the complexity of early word segmentation, and highlight the importance of including contextual factors in developmental models of word segmentation.

A Hybrid Model for Chinese Word Segmentation

This paper describes a hybrid model that combines machine learning with linguistic and statistical heuristics for integrating unknown word identification with Chinese word segmentation. The model consists of two major components: a tagging component that annotates each character in a Chinese sentence with a position-of-character (POC) tag that indicates its position in a word, and a merging component that transforms a POC-tagged character sequence into a word-segmented sentence. The tagging component uses a support vector machine (Vapnik, 1995) based tagger to produce an initial tagging of the text and a transformation-based tagger (Brill, 1995) to improve the initial tagging. In addition to the POC tags assigned to the characters, the merging component incorporates a number of linguistic and statistical heuristics to detect words with regular internal structures, recognize long words, and filter non-words. Experiments show that, without resorting to a separate unknown word identification mechanism, the model achieves an F-score of 95.0% for word segmentation and a competitive recall of 74.8% for unknown word identification.

A study on word-based and integral-bit Chinese text compression algorithms

Experimental results show that a word-based arithmetic coding scheme can achieve a higher compression performance for Chinese text. However, an arithmetic coding scheme is a fractional-bit compression algorithm which is known to be time consuming. In this article, we change the direction to study how to cascade the word segmentation model with a faster alternative, the integral-bit compression algorithm. It is shown that the cascaded algorithm is more suitable for practical usage. Among several word-based integral-bit compression algorithms, WLZSSHUF achieves the best compression results. Not only can it achieve a comparable compression ratio with a PPM compressor, COMP-2, it demonstrates a faster compression and decompression speed. In the last part of this article, the relation between the accuracy of the word segmentation model (match ratio) and the performance of the compression algorithm (compression ratio) are analyzed. By varying the match ratio, it was discovered that the growth rate of the compression ratio is content-dependent and close to linear. The results of our study will help the practitioners of information retrieval to design word-based compression algorithms for Chinese. This is particularly useful to multilingual digital libraries in which a massive volume of data is often involved.

PARSER: A Model for Word Segmentation

Saffran, Newport, and Aslin (1996b) showed that adults were able to segment into words an artificial language that included no pauses or other prosodic cues for word boundaries. We propose an account of their results that requires only limited computational abilities and memory capacity. In this account, parsing emerges as a natural consequence of the on-line attentional processing of the input, thanks to basic laws of memory and associative learning. Our account was implemented in a computer program, PARSER. Simulations revealed that PARSER extracted the words of the language well before exhausting the material presented to participants in the Saffran et al. experiments. In addition, PARSER was able to simulate the results obtained under attention-disturbing conditions (Saffran, Newport, Aslin, Tunick, & Barrueco, 1997) and those collected from 8-month-old infants (Saffran, Aslin, and Newport, 1996a). Finally, the good performance of PARSER was not limited to the trisyllabic words used by Saffran et al., but also extended to a language composed of one- to five-syllable words.

The role of stress in word segmentation

Research reported at a previous meeting of the Acoustical Society examined the role of strong syllables in segmentation [P. N. Yerkey and J. R. Sawusch, J. Acoust. Soc. Am. 91, 2338(A) (1992)]. Subjects were asked to monitor bisyllabic nonwords for the presence of a word. The stimuli initially contained two stressed syllables, and were later edited to reduce the amplitude and duration of the second syllable. The results showed no effect of vowel quality on segmentation. This suggests that the vowel was not influencing segmentation, but that stress alone served as the primary cue for segmentation. In the present studies, the experimental items were recorded with naturally stressed and unstressed second syllables. In different experiments, the stimuli were presented to listeners with their natural stress patterns or after alteration of the amplitude and duration of the second syllable. The pattern of results across variations in second syllable stress and vowel quality (tense, lax, or neutral) will be discussed with respect to its influence on perception and implications for models of word segmentation. [Work supported by NIDCD Grant No. DC 00219 to SUNY at Buffalo.]