Annotating Socio-Cultural Structures in Text

Abstract

Abstract : This report details the development of the Machine Learning for Text Annotation Workbench (MaLTAW), an annotation assistance tool that incorporates automated classification to reduce the difficulty of manual annotation. The text corpus is large and consists of numerous reports, lessons learned, and best practices. There are several technical challenges posed by the text corpus and the annotation process. The document set is complex due to the size of the documents, the variety of formats and the range of subject matter. The annotation taxonomy is large and unstructured and may be applied to the text body without constraints. Consequently, the search space for the label(s) become prohibitively large and it becomes necessary to adopt strategies that reduce the complexity of the classification process. A simplification technique is introduced to reduce the large classification search space. Precision is improved by supplementing these predictive algorithms with similarity-based measures. MaLTAW is evaluated for performance using both prediction-based metrics and ranking-based metrics. It is shown that MaLTAW performs better than a competing algorithm on all evaluation metrics.