Bayesian probabilistic forecasting with large-scale educational trend data: a case study using NAEP

David Kaplan,Mingya Huang

doi:10.1186/s40536-021-00108-2

David Kaplan, Mingya Huang

Open Access

https://doi.org/10.1186/s40536-021-00108-2

Copy DOI

Abstract

Of critical importance to education policy is monitoring trends in education outcomes over time. In the United States, the National Assessment of Educational Progress (NAEP) has provided long-term trend data since 1970; at the state/jurisdiction level, NAEP has provided long-term trend data since 1996. In addition to the national NAEP, all 50 states and United States jurisdictions participate in the state NAEP assessment. Thus, NAEP provides important monitoring and forecasting information regarding population-level academic performance of relevance to national and international goals. However, an inspection of NAEP trend reports shows that relatively simple trend plots are provided; and although these plots are important for communicating general trend information, we argue that much more useful information can be obtained by adopting a Bayesian probabilistic forecasting point of view. The purpose of this paper is to provide a Bayesian probabilistic forecasting workflow that can be used with large-scale assessment trend data generally, and to demonstrate that workflow with an application to the state NAEP assessments.

Highlights

Of critical importance to education policy is monitoring trends in education outcomes over time
We find that the log predictive score (LPS) and Kullback–Leibler divergence (KLD) values within model prior settings are virtually identical across all parameter priors
Summary and conclusions In this paper, we provided a workflow that permits Bayesian probabilistic forecasting to be applied to large-scale assessments, with National Assessment of Educational Progress (NAEP) being used as an example

Summary

Introduction

Of critical importance to education policy is monitoring trends in education outcomes over time. As will be discussed below, our focus will be on time-invariant outcomes because we are interested in modeling the average growth rate across states and not time-varying features of the growth trajectories Another important flexibility in latent variable growth curve modeling allows estimation of non-linear trajectories using latent basis methods. Of importance to this paper is that theory and applications of Bayesian model averaging have shown that it provides better long-run predictive performance to that of any single model based on a class of scoring rules used in probabilistic forecasting analysis (Raftery et al, 1997). The estimated growth rate without predictors is compared to predicted growth rate using Bayesian model averaging along with different choices of model and parameter priors.

Q2 yields “g-RIC”-shrinkage factor where

GirlsEnrollDiff

FTEdiff2

Findings

ReadDiff