Abstract
We outline a method of detecting ad hoc, or anomalous, rules in treebank grammars, by exploiting the fact that such rules do not fit with the rest of the grammar. Ad hoc rules are rules used for specific constructions in one data set and unlikely to be used again. These include ungeneralizable rules, erroneous rules, rules for ungrammatical text, and rules which are not consistent with the rest of the annotation scheme. Based on the idea that valid rules should receive support from other rules in the grammar, we develop two methods for detecting ad hoc rules in flat treebanks and show they are successful in detecting such rules. Although one can put some linguistic knowledge into determining rule similarity and dissimilarity, the methods work best by using a simple, modified Levenshtein distance. We illustrate this on the English Wall Street Journal treebank and the German TIGER treebank. For the latter, we extend the method to formalisms incorporating discontinuous constituents, employing CFG-like rules for the comparisons.
Highlights
Our starting point for comparing rules comes from a method of annotation error detection which searches for inconsistency of labeling within local trees (Dickinson and Meurers, 2005b)
We have presented work on detecting ad hoc rules in treebanks, where an ad hoc rule is an annotation error, covers an ungrammatical sentence, reveals issues with the uniformity of an annotation scheme or is a rule that does not generalize well
We started with a notion of equivalence classes, the idea that different rules express the same linguistic content, with respect to valency, and moved on to more general similarity metrics
Summary
When extracting rules from treebanks, especially constituency-based treebanks employing flat structures, grammars often limit the set of rules (e.g., Charniak, 1996), due to the large number of rules (Krotov et al, 1998) and “leaky” rules that can lead to mis-analysis (Foth and Menzel, 2006). When ungrammatical or non-standard text is used, treebanks employ rules to cover it, but do not usually indicate ungrammaticality in the annotation These rules are only to be used in certain situations, e.g., for typographical conventions such as footnotes, and pose a problem if the set of treebank rules is intended to accurately capture the grammar of a language. This is true in the case of precision grammars for grammar checking and generation (e.g., Wagner et al, 2007, Bender et al, 2004), and in applications like intelligent computer-aided language learning, where learner input is parsed to detect what is correct or not (e.g., Metcalf and Boyd, 2006, Dickinson and Lee, 2009). We here highlight the theoretical issues involved across different scenarios and in many parts provide more extensive evaluation
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
