Text disambiguation by finite state automata, an algorithm and experiments on corpora

Abstract

Consulting a dictionary for the words of a given text provides multiple solutions, that is, ambiguities; thus, the sequence of words pilot studies could lead for example to:pilot: N singular, V infinitive, V (conjugated)studies: N plural, V (conjugated)pilot studies: N plural (compound).These informations could be organized in the form of a finite automaton such as:[see pdf for figure]The exploration of the context should provide clues that eliminate the non-relevant solutions. For this purpose we use local grammar constraints represented by finite automata. We have designed and implemented an algorithm which performs this task by using a large variety of linguistic constraints. Both the texts and the rules (or constraints) are represented in the same formalism, that is finite automata. Performing subtraction operations between text automata and constraint automata reduce the ambiguities. Experiments were performed on French texts with large scale dictionaries (one dictionary of 600.000 simple inflected forms and one dictionary of 150.000 inflected compunds). Syntactic patterns represented by automata, including shapes of compound nouns such as Noun followed by an Adjective (in gender-number agreement) (Cf 5.1), can be matched in texts.This process is thus an extension of the classic matching procedures because of the on-line dictionary consultation and because of the grammar constraints. It provides a simple and efficient indexing tool.