Can diagnostic tests help identify model misspecification in integrated stock assessments?

Abstract

A variety of data types can be included in contemporary integrated stock assessments to simultaneously provide information on all estimated parameters. Conflicts between data, which are often a symptom of model misspecification and evident as model misfit, can affect the estimates of important parameters and derived quantities. Unfortunately, there are few standard diagnostic tools available for integrated stock assessment models that can provide the analyst with all the information needed to determine if there is substantial model misspecification. In this study, we use simulation methods to evaluate the ability of commonly-used and recently-proposed diagnostic tests to detect model misspecification in the observation model process (i.e., the incorrect form for survey selectivity), systems dynamics (i.e., incorrect assumed values for steepness of the stock-recruitment relationship and natural mortality), and incorrect data weighting. The diagnostic tests evaluated here were: i) residuals analysis (SDNR and runs test); ii) retrospective analysis; iii) the R0 likelihood component profile; iv) the age-structured production model (ASPM); and v) catch-curve analysis (CCA). The efficacy of the diagnostic tests depended on whether the misspecification was in the observation or systems dynamics model. Residual analyses were easily the best detector of misspecification of the observation model while the ASPM test was the only good diagnostic for detecting misspecification of system dynamics model. Retrospective analysis and the R0 likelihood component profile infrequently detected misspecified models, and CCA had a high probability of rejecting correctly-specified models. Finally, applying multiple carefully selected diagnostics can increase the power to detect misspecification without substantially increasing the probability of falsely concluding there is misspecification when the model is correctly specified.