A Weighted Accuracy Measure for Land Cover Mapping: Comment on Johnson et al. Local Climate Zone (LCZ) Map Accuracy Assessments Should Account for Land Cover Physical Characteristics that Affect the Local Thermal Environment. Remote Sens. 2019, 11, 2420

Benjamin Bechtel,Iain D Stewart,Matthias Demuzere

doi:10.3390/rs12111769

Benjamin Bechtel, Iain D Stewart + Show 1 more

Open Access

https://doi.org/10.3390/rs12111769

Copy DOI

Journal: Remote sensing	Publication Date: May 31, 2020
Citations: 20	License type: CC BY 4.0

Affiliation: Ruhr University Bochum, University of Toronto

Abstract

In multi-class classification tasks such as land cover mapping, the achieved accuracies inherently depend on the complexity of the class typology. More specifically, the more complex the typology of (land cover) classes, the lower the resulting accuracies, since the common measures only consider whether a sample was correctly classified or not. To overcome this, a weighted accuracy measure was introduced in 2017 for the case of Local Climate Zone (LCZ) mapping. This method was recently criticized by Johnson and Jozdani and an alternative method was proposed. In this comment, we explain the weighted accuracy measure in more detail and reject the criticism. We show that the proposed method of Johnson and Jozdani is based on weakly supported assumptions. In addition, it is argued that the weighted accuracy is potentially a useful complementary measure beyond the LCZ classification case.

Highlights

It is a general problem in land cover mapping and other multi-class classification tasks that the achieved accuracies are a function of the complexity—or in other words semantic resolution—of the class typology
It was highlighted that a weighted accuracy is always related to a specific purpose, and its appropriateness requires expert judgement
The suggested accuracy measure of JJ19 was rejected by showing that it lacks innovation and involves wrong assumptions

Summary

Introduction

It is a general problem in land cover mapping and other multi-class classification tasks that the achieved accuracies are a function of the complexity—or in other words semantic resolution—of the class typology. In their recent technical note, Johnson and Jozdani [3] (hereafter JJ19) argued that Local Climate Zone (LCZ) mapping accuracies should account for the land cover characteristics that affect the physical environment While this claim is consistent with the considerations above, the method itself is not innovative and based on wrong assumptions. While they correctly acknowledge that the idea has already been proposed by Bechtel et al [4], their interpretation of the original method is mistaken They claim that the original “idea of weighting the error matrix has merit, but their proposed methodology had two main shortcomings: (1) the determined weights were subjective and not fully transparent (e.g., no equations provided to explain how they were derived); and (2) the weights incorrectly applied a higher penalty to the misclassification of more physically similar LCZ types.”. We show why the JJ19 paper is mistaken and based on wrong assumptions, and why their metric is not compelling

The Weighted Accuracy

Construction of the LCZ Metric

Conclusions