Keynote: Adventures in Annotation: Providing High Quality Labels for Supervised Machine Learning

Abstract

Ground truth annotation is a crucial prerequisite for the application of supervised classification methods for both, training and evaluation. To this end, the quality of the labels plays a significant role for the final ability of the trained system to recognize the concepts of interest. This talk addresses the problem of creating high quality labels by manual annotation and their application in downstream classification tasks. To this end, three studies will be reviewed that relied on the quality of the collected ground truth annotation and the procedure for ground truth annotation will be discussed. The first study applies planning operators and inverse planning to ensure that annotations of human behavior are causally correct [1] and, thus, can be used for downstream plan and intention recognition [2] . In the second study, an annotation scheme is derived from a clinical observation tool [3] and used as ground truth labels for the automated detection of challenging behavior in advanced stages of dementia [4] . In the third study, the annotation of named entities and their relations is discussed [5] that serves as reliable ground truth annotation for large scale information extraction [6] . Finally, the key factors of all studies will be summarized and discussed.