On the influence of training data quality on text document classification using machine learning methods

Abstract

The main target of this paper was to study the influence of training data quality on the text document classification performance of machine learning methods. A graded relevance corpus of ten classes and 957 text documents was classified with Self-Organising Maps (SOMs), learning vector quantisation, k-nearest neighbours searching, naïve Bayes and support vector machines. The relevance level of a document (irrelevant, marginally, fairly or highly relevant) was used as a measure of the quality of the document as a training example, which is a new approach. The classifiers were evaluated with micro- and macro-averaged classification accuracies. The results suggest that training data of higher quality should be preferred, but even low-quality data can improve a classifier, if there is plenty of it. In addition, further means to facilitate classification by the SOMs were explored. The novel set of SOM approach performed clearly better than the original SOM and comparably against supervised classification methods.