Combining imperfect automated annotations of underwater images with human annotations to obtain precise and unbiased population estimates

Abstract

Optical methods for surveying populations are becoming increasingly popular. These methods often produce hundreds of thousands to millions of images, making it impractical to analyze all the images manually by human annotators. Computer vision software can rapidly annotate these images, but their error rates are often substantial, vary spatially and are autocorrelated. Hence, population estimates based on the raw computer automated counts can be seriously biased. We evaluated four estimators that combine automated annotations of all the images with manual annotations from a random sample to obtain (approximately) unbiased population estimates, namely: ratio, offset, and linear regression estimators as well as the mean of the manual annotations only. Each of these estimators was applied either globally or locally (i.e., either all data were used or only those near the point in question, to take into account spatial variability and autocorrelation in error rates). We also investigated a simple stratification scheme that splits the images into two strata, based on whether the automated annotator detected no targets or at least one target. The 16 methods resulting from a combination of four estimators, global or local estimation, and one stratum or two strata, were evaluated using simulations and field data. Our results indicated that the probability of a false negative is the key factor determining the best method, regardless of the probability of false positives. Stratification was the most effective method in improving the accuracy and precision of the estimates, provided the false negative rate was not too high. If the probability of false negatives is low, stratified estimation with the local ratio estimator or local regression (essentially geographically weighted regression) is best. If the probability of false negatives is high, no stratification with a simple global linear regression or simply the manual sample mean alone is recommended.