Thematic Classification Accuracy Assessment with Inherently Uncertain Boundaries: An Argument for Center-Weighted Accuracy Assessment Metrics

Aaron E Maxwell,Timothy A Warner

doi:10.3390/rs12121905

Abstract

Accuracy assessment is one of the most important components of both applied and research-oriented remote sensing projects. For mapped classes that have sharp and easily identified boundaries, a broad array of accuracy assessment methods has been developed. However, accuracy assessment is in many cases complicated by classes that have fuzzy, indeterminate, or gradational boundaries, a condition which is common in real landscapes; for example, the boundaries of wetlands, many soil map units, and tree crowns. In such circumstances, the conventional approach of treating all reference pixels as equally important, whether located on the map close to the boundary of a class, or in the class center, can lead to misleading results. We therefore propose an accuracy assessment approach that relies on center-weighting map segment area to calculate a variety of common classification metrics including overall accuracy, class user’s and producer’s accuracy, precision, recall, specificity, and the F1 score. This method offers an augmentation of traditional assessment methods, can be used for both binary and multiclass assessment, allows for the calculation of count- and area-based measures, and permits the user to define the impact of distance from map segment edges based on a distance weighting exponent and a saturation threshold distance, after which the weighting ceases to grow. The method is demonstrated using synthetic and real examples, highlighting its use when the accuracy of maps with inherently uncertain class boundaries is evaluated.

Highlights

Assessment of the accuracy of maps produced through thematic classification is a central component of remote sensing, and has further applications in pattern recognition and computer vision [1,2,3,4,5,6,7]
Rigorous and consistent comparison of classification products, methods, and algorithms requires well-defined and appropriate metrics [4,9,10,11]. Such metrics generally assume that map features have discrete and well-defined boundaries, and that the true value of all pixels can be ascertained with equal accuracy, regardless of spatial location relative to feature edges
This study proposes and demonstrates an assessment method that is appropriate for assessing the mapping of features with inherently uncertain boundaries using feature center-weighted assessment metrics generated from an error matrix

Summary

Introduction

Assessment of the accuracy of maps produced through thematic classification is a central component of remote sensing, and has further applications in pattern recognition and computer vision [1,2,3,4,5,6,7]. Rigorous and consistent comparison of classification products, methods, and algorithms requires well-defined and appropriate metrics [4,9,10,11]. Such metrics generally assume that map features have discrete and well-defined boundaries, and that the true value of all pixels can be ascertained with equal accuracy, regardless of spatial location relative to feature edges. Inherently uncertain boundaries are common in natural, real-world landscapes Ignoring this uncertainty may produce a pessimistic bias in the resulting accuracy estimates [5]. Pontius and Millones [22], who advocate against the use of Kappa, introduced the alternative measures of quantity and allocation disagreement

Methods

Findings

Discussion

Conclusion