Japanese Morphological Analysis Research Articles

Development of a computer-assisted Japanese functional expression learning system for Chinese-speaking learners

Because a large number of Chinese characters are commonly used in both Japanese and Chinese, Chinese-speaking learners of Japanese as a second language (JSL) find it more challenging to learn Japanese functional expressions than to learn other Japanese vocabulary. To address this challenge, we have developed Jastudy, a computer-assisted language learning (CALL) system designed specifically for Chinese-speaking learners studying Japanese functional expressions. Given a Japanese sentence as an input, the system automatically detects Japanese functional expressions using a character-based bidirectional long short-term memory with a conditional random field (BiLSTM-CRF) model. The sentence is then segmented and the parts of speech (POS) are tagged (word segmentation and POS tagging) by a Japanese morphological analyzer, MeCab ( http://taku910.github.io/mecab/ ), trained using a CRF model. In addition, the system provides JSL learners with appropriate example sentences that illustrate Japanese functional expressions. The system uses a ranking system, which gives easier sentences a higher rank, when selecting example sentences. A support vector machine for ranking (SVMRank) algorithm estimates the readability of example sentences, using Japanese-Chinese common words as an important feature. A k-means clustering algorithm is used to cluster example sentences that contain functional expressions with the same meanings, based on part-of-speech, conjugation form, and semantic attributes. Finally, to evaluate the usefulness of the system, we have conducted experiments and reported on a preliminary user study involving Chinese-speaking JSL learners.

A comparative study of dictionaries and corpora as methods for language resource addition

In this paper, we investigate the relative effect of two strategies for language resource addition for Japanese morphological analysis, a joint task of word segmentation and part-of-speech tagging. The first strategy is adding entries to the dictionary and the second is adding annotated sentences to the training corpus. The experimental results showed that addition of annotated sentences to the training corpus is better than the addition of entries to the dictionary. In particular, adding annotated sentences is especially efficient when we add new words with contexts of several real occurrences as partially annotated sentences, i.e. sentences in which only some words are annotated with word boundary information. According to this knowledge, we performed real annotation experiments on invention disclosure texts and observed word segmentation accuracy. Finally we investigated various language resource addition cases and introduced the notion of non-maleficence, asymmetricity, and additivity of language resources for a task. In the WS case, we found that language resource addition is non-maleficent (adding new resources causes no harm in other domains) and sometimes additive (adding new resources helps other domains). We conclude that it is reasonable for us, NLP tool providers, to distribute only one general-domain model trained from all the language resources we have.

日本語形態素解析における未知語処理の一手法―既知語から派生した表記と未知オノマトペの処理―

This paper presents a simple but effective approach to unknown word processing in Japanese morphological analysis, which handles 1) unknown words that are derived from words in a pre-defined lexicon and 2) unknown onomatopoeias. Our approach leverages derivation rules and onomatopoeia patterns, and correctly recognizes certain types of unknown words. Experiments revealed that our approach recognized about 4,500 unknown words in 100,000 Web sentences with only 80 harmful side effects and a 6% loss in speed.

絵本のテキストを対象とした形態素解析

これまで，主に新聞などのテキストを対象とした解析では，形態素解析器を始めとして高い解析精度が達成されている．しかし分野の異なるテキストに対しては，既存の解析モデルで，必ずしも高い解析精度を得られるわけではない．そこで本稿では，既存の言語資源を対象分野の特徴にあわせて自動的に変換する手法を提案する．本稿では，絵本を解析対象とし，既存の言語資源を絵本の特徴にあわせて自動的に変換し，学習に用いることで相当な精度向上が可能であることを示す．学習には既存の形態素解析器の学習機能を用いる．さらに，絵本自体にアノテーションしたデータを学習に用いる実験を行い，提案手法で得られる効果は，絵本自体への約 11,000 行，90,000 形態素のアノテーションと同程度であることを示す．また，同じ絵本の一部を学習データに追加する場合と，それ以外の場合について，学習曲線や誤り内容の変化を調査し，効果的なアノテーション方法を示す．考察では，絵本の対象年齢と解析精度の関係や，解析精度が向上しにくい語の分析を行い，更なる改良案を示す．また，絵本以外への適用可能性についても考察する．

USABILITY TESTING OF JAPANESE CAPTIONS SEGMENTATION SYSTEM TO SCAFFOLD BEGINNERS TO COMPREHEND JAPANESE VIDEOS

A major learning difficulty of Japanese foreign language (JFL) learners is the complex composition of two syllabaries, hiragana and katakana, and kanji characters adopted from logographic Chinese ones. As the number of Japanese language learners increases, computer-assisted Japanese language education gradually gains more attention. This study aimed to adopt a Japanese word segmentation system to help JFL learners overcome literacy problems. This study adopted MeCab, a Japanese morphological analyzer and part-of-speech (POS) tagger, to segment Japanese texts into separate morphemes by adding spaces and to attach POS tags to each morpheme for beginners. The participants were asked to participate in three experimental activities involving watching two Japanese videos with general or segmented Japanese captions and complete the Nielsen’s Attributes of (NAU) survey and the After Scenario Questionnaire (ASQ) to evaluate the usability of the learning activities. The results of the system evaluation showed that the videos with the segmented captions could increase the participants’ learning motivation and willingness to adopt the word segmentation system to learn Japanese. To cite this document: Ya-Fei Yang, Ching-Ju Chao, Yu-Ling Lin, and Chih-Kai Chang, Usability testing of Japanese captions segmentation system to scaffold beginners to comprehend Japanese videos, International Journal of Cyber Society and Education, Vol. 6, No. 1, pp. 1-14, 2013. Permanent link to this document: http://dx.doi.org/10.7903/ijcse.1083

日本語形態素解析とその周辺領域における最近の研究動向

日本語形態素解析とその周辺領域における最近の研究動向

Best practices and tools for personal information compliance management

Recent incidents involving the loss of personal information and identity theft have raised concerns worldwide over information privacy. In Japan, the Personal Information Protection Act went into effect in April 2005, requiring every enterprise to manage sensitive personal information on servers, workstations, and personal computers throughout the organization. This paper describes two tools we developed to assist in the management of personal information, aDesigner and the Personal Information Detection (PID) tool. The aDesigner tool scans an entire Web site to determine if each HTML page complies with the IBM privacy guidelines for external Web sites. PID is capable of automatically identifying “named entities,” such as personal names, addresses, or telephone numbers in the textual parts of target files based on Japanese morphological analysis technology. This paper also summarizes the best practices used in IBM Japan for privacy management and presents statistical results concerning personal information gathered through deployment of these tools.

Use of morphological analysis in protein name recognition

Protein name recognition aims to detect each and every protein names appearing in a PubMed abstract. The task is not simple, as the graphic word boundary (space separator) assumed in conventional preprocessing does not necessarily coincide with the protein name boundary. Such boundary disagreement caused by tokenization ambiguity has usually been ignored in conventional preprocessing of general English. In this paper, we argue that boundary disagreement poses serious limitations in biomedical English text processing, not to mention protein name recognition. Our key idea for dealing with the boundary disagreement is to apply techniques used in Japanese morphological analysis where there are no word boundaries. Having evaluated the proposed method with GENIA corpus 3.02, we obtain F-measure of 69.01 on a strict criterion and 79.32 on a relaxed criterion. The result is comparable to other published work in protein name recognition, without resorting to manually prepared ad hoc feature engineering. Further, compared to the conventional preprocessing, the use of morphological analysis as preprocessing improves the performance of protein name recognition and reduces the execution time.

農業関連文書用形態素解析サーバの開発とテキストの自動分類による検証

農業関連文書用形態素解析サーバの開発とテキストの自動分類による検証

メタデータブラウザの多言語対応に向けた課題への取り組み

The Research Center for Knowledge Communities at University of Tsukuba introduced the Knowledge Community Information System (KCIS) in February 2003. KCIS inherited the metadata of the digital library system of the University of Library and Information Science (ULIS-DL). In KCIS, metadata are able to be written in several languages because they are expressed in XML and Unicode. The browsing system of metadata of ULIS-DL were designed to handle metadata written in Japanese. This paper discusses problems of applying the browsing system to multilingual metadata. One of the problems is that the system extracts words from metadata with Japanese morphological analyzer. Another problem is synonyms between different languages. This paper describes a prototype system which are developed to make the issues clear. It also describes development of a experimental system to display multilingual XML documents. It is needed for using the browsing system of metadata with lightweight terminals.

Mark-up-Based Writing Error Analysis Model in an On-line Classroom

This paper describes a new component called Writing Error Analysis Model (WEAM) in the CoCoA system (Ogata et al., 1998) for instructing composition of writing in Japanese as a foreign language (JFL). The WEAM can be used for analyzing learners' writing error in certain types of morphological errors and selecting the appropriate compositions for learners' revising exercises. In accordance with the classifications of morphological errors that are often made by JFL learners (Chino & Nishina, 1977; Ichikawa, 1997), a Japanese morphological analysis system named Chasen (Matsumoto et al., 1997) is applied in the WEAM for recognizing and tabulating the selected 33 different types of morphological errors. The similarity of morphological errors between the error patterns in JFL documents stored in the CoCoA's database and the method of calculation are proposed in WEAM for retrieving documents whose sentences can provide useful exercise material for students.

A generalized algorithm for Japanese morphological analysis and a comparative evaluation of some heuristics

AbstractIn ordinary written Japanese, words are not separated by spaces. Therefore morphological analysis involves segmenting and tagging sentences. Since each sentence has a huge number of possible tagged segmentations, various criteria have been proposed for making plausible decisions. However, there are still no unified frameworks that incorporate various heuristics, and there has been no comparative evaluation of commonly used heuristics. This paper presents a clear framework to describe various heuristics, and an N‐best algorithm for extracting optimal solutions. The time complexity of this algorithm is O(nNlog2(1 + N)), where n is the sentence length. The advantage of the N‐best algorithm over the standard beam search algorithm is also discussed. This paper also presents a comparative evaluation of three major heuristics, and proposes a precise and portable rule‐based heuristic. Estimation was done using the aforementioned algorithm and six criteria. The newly proposed heuristic is based upon the Extended Least Bunsetsu (Phrase) Number method.