A word segmentation method with dynamic adapting to text using inductive learning

Abstract

We have proposed a method of word segmentation for non-segmented language using Inductive Learning. This method uses only surface information of a text, so that it has an advantage that is entirely not dependent on any specific language. In this method, we consider that a character string of appearing frequently in a text has a high possibility as a word. The method predicts unknown words by recursively extracting common character strings. With the proposed method, the segmentation results can adapt to different users and fields. To evaluate effectivety for Chinese word segmentation and adaptability for different fields, we have done the evaluation experiment with Chinese text of the two fields.