Email Answering by Matching Question and Context-Specific Text Patterns: Performance and Error Analysis

Abstract

Automated answering of frequent email inquiries is a text categorization task with narrow text categories, where all messages in one text category have the same answer. Such email categorization is optimized for high precision and at least acceptable recall. We tested matching of surface text patterns to nearly ten thousand email messages and achieved around 90 % precision; the corresponding recall figures were 45–75 % in different text categories. In order to achieve this performance level, the text patterns are designed to identify both the context of an email inquiry and the actual need that has created the inquiry—a question, request, or complaint. Our error analysis has pinpointed 12 reasons why text pattern matching may fail.