Overselling overall map accuracy misinforms about research reliability

Guofan Shao,Jiangfu Liao,Lina Tang

doi:10.1007/s10980-019-00916-6

Guofan Shao, Jiangfu Liao + Show 1 more

Open Access

https://doi.org/10.1007/s10980-019-00916-6

Copy DOI

Abstract

ContextImage classification is routine in a variety of disciplines, and analysts rely on accuracy metrics to evaluate the resulting maps. The most frequently used accuracy metric in Earth resource remote sensing is overall accuracy. However, the inherent properties of this accuracy metric make it inappropriate as the single metric for map assessment, particularly when a map contains imbalanced categories.ObjectivesWe discuss four noteworthy problems with overall accuracy. Under circumstances frequently encountered, overall accuracy is misleading or misinterpreted.MethodsLiterature review, hypothetical examples, and mathematic equations are used to prove overall accuracy is a poor general indicator of map quality.ConclusionsAny research that involves classification techniques or a map product that is evaluated only with overall accuracy may be unreliable. It is necessary for map providers to publish the error matrix and its development procedure so that map users can computer whatever metrics as they wish.

Highlights

It is necessary for map providers to publish the error matrix and its development procedure so that map users can computer whatever metrics as they wish
If only a single accuracy metric is reported for a classification map, it Reference data j =1 j=2 j=J
Overselling overall accuracy should be discouraged because the O metric has inherent properties that lead to misleading perceptions about the soundness of classification methods and map-based research

Summary

Introduction

Digital classification is commonly used to convert image data into map products by assigning each pixel into one of two or more categories. One standard way to evaluate maps is to use error matrices ( known as confusion matrices) to compute accuracy metrics or indices, including map-level and category-level accuracies. The overall accuracy is the percentage of cases (e.g., pixels or sampling units) correctly classified.

Objectives

Results

Conclusion