Review of methodologies for detecting an observer effect in commercial fisheries data

Abstract

Observers are deployed on commercial fishing trips to collect representative samples of discard rates. However, fishers may change their fishing habits when an observer is onboard (“observer effect”) or observer programs may over- or under-sample portions of the fleet (“deployment effect”). If the extent of these effects are substantial, observer data will not be representative of unobserved trips, potentially biasing the estimation of discards. This sampling bias can impact catch monitoring, stock assessments, and fishery management. The purpose of this study was to examine the power and error rate of several published methods for detecting an observer effect using a simulation of observer and deployment effects at varying sampling ratios (i.e., observer coverage) for several sample statistics. The simplest methods (t-test and F-test for difference of means and variances) provided an accurate although imprecise estimate of the observer effect size, but only when there were no deployment effects. A generalized linear mixed effects model (GLMM) was also not reliable for detecting small bias, but was not confounded by deployment effects and was relatively robust to changing coverage rates. The most complicated tests involved comparing differences in trip characteristics between subsequent trips for observed-unobserved and unobserved-unobserved pairs. These tests were able to detect smaller observer effects and were not confounded by deployment effects, but were unreliable at high coverage rates (>60%), producing both high false positive and false negative rates. Sensitivity tests also showed differing detection accuracy as the distribution of the metric of interest changed. Thus, the optimal test for detecting an observer effect will depend on the metric of interest, the coverage rate, and whether a deployment effect exists. An example from the New England groundfish fishery is provided to illustrate how conflicting results may be explained. Results should always be considered carefully when declaring that an observer effect is or is not occurring because of the sensitivity of the tests.