Innovations in text interpretation

Abstract

The field of natural language processing is developing a new concentration on interpreting extended texts, with applications in information retrieval, text categorization, and data extraction. The research that addresses these problems represents the first real task-driven focus since machine translation research in the 1960s. Text interpretation applications have already produced good results in accuracy and throughput. This new focus on task-driven text interpretation has been the driving force for a number of advances in the field, because earlier systems fell so far short of the coverage required to interpret bodies of text. The innovations behind this scale-up include work in lexicon development and representation, weak methods of corpus analysis and text pre-processing, and flexible control architectures for parsing. Together, these methods provide coverage and accuracy in interpretation by extending the knowledge that a system can use and controlling how this knowledge is applied. This paper explains the context in which this research is conducted, along with the general progress of the field and some of the details of how our own system realizes these advances.