An Evaluation of the Forecasting Accuracy of Two Trend Estimation Methods from Two Computer Adaptive Tests

Abstract

This study evaluated the forecasting accuracy of trend estimation methods applied to time-series data from computer adaptive tests (CATs). Data were collected roughly once a month over the course of a school year. We evaluated the forecasting accuracy of two regression-based growth estimation methods (ordinary least squares and Theil-Sen). The precision, or accuracy, of predictions were heavily influenced by how far performance was forecasted into the future (1, 3, or 5 months) and the number of observations available to estimate growth (3, 4, or 5). When performance was forecasted further and growth estimates were based upon fewer observations, predictions were off by as much as two times the average conditional standard error of measurement for a grade level. No combinations of data collections schedule, forecasting length, or estimation method led to consistent bias in predictions. Educators should be cautious when using trend-line decision rules to predict future performance. Suggestions for alternate decision rules to explore with CATs are offered.