Sources of error in accuracy assessment of thematic land-cover maps in the Brazilian Amazon

Abstract

Valid measures of map accuracy are critical, yet can be inaccurate even when following well-established procedures. Accuracy assessment is particularly problematic when thematic classes lie along a land-cover continuum, and boundaries between classes are ambiguous. In this study, we examined error sources introduced during accuracy assessment of a regional land-cover map generated from Landsat Thematic Mapper (TM) data in Rondônia, southwestern Brazil. In this dynamic, highly fragmented landscape, the dominant land-cover classes represent a continuum from pasture to second growth to primary forest. We used high spatial resolution, geocoded videography as a reference, and focused on second-growth forest because of its potential contribution to the regional carbon balance. To quantify subjectivity in reference data labeling, we compared reference data produced by five trained interpreters. We also quantified the impact of other error sources, including geolocation errors between the map and reference data, land-cover changes between dates of data collection, heterogeneous reference samples, and edge pixels. Interpreters disagreed on classification of almost 30% of the samples; mixed reference samples and samples located in transitional classes accounted for a majority of disagreements. Agreement on second-growth forest labels between any two interpreters averaged below 50%, while agreement on primary forest was over 90%. Greater than 30% of disagreement between map and reference data was attributed to geolocation error, and 2.4% of disagreement was attributed to change in land cover between dates. After geocorrection, 24% of remaining disagreements corresponded to reference samples with mixed land cover, and 47% corresponded to edge pixels on the classified map. These findings suggest that: (1) labels of continuous land-cover types are more subjective and variable than commonly assumed, especially for transitional classes; however, using multiple interpreters to produce the reference data classification increases reference data accuracy; and (2) validation data sets that include only non-mixed, non-edge samples are likely to result in overly optimistic accuracy estimates, not representative of the map as a whole. These results suggest that different regional estimates of second-growth extent may be inaccurate and difficult to compare.