Boosting qualifies capture–recapture methods for estimating the comprehensiveness of literature searches for systematic reviews

Abstract

Objective Capture–recapture methods were proposed to evaluate the comprehensiveness of systematic literature searches. We investigate the statistical feasibility of capture–recapture techniques with model selection for estimating the number of missing references in literature searches using two systematic reviews in gastroenterology and hematology. Study Design and Setting First, we compared manually selected Poisson regression models that differ with respect to included interactions. Secondly, we performed selection via componentwise boosting, which provides automatic variable selection. The proposed boosting technique is a regularized, stepwise procedure allowing to distinguish between mandatory and optional variables. Results from all models were compared based on Akaike’s Information Criterion and the Bayesian Information Criterion. Results For the first example, the best manually selected model suggested a number of 82 missing articles (95% CI: 52–128), whereas the boosting technique provided 127 (95% CI: 86–186) missing articles. For the second example, 140 (95% CI: 116–168) missing articles were estimated for the manually selected and 188 (95% CI: 159–223) for the automatically selected model. Conclusion Capture–recapture analysis requires the selection of an appropriate model. Because of problems of variable selection and overfitting, manual model selection yielded large estimates, varying markedly, with broad confidence intervals. By contrast, boosting was robust against overfitting and automatically created an appropriate model for inference.