Abstract
Bayesian estimation of multidimensional item response theory (IRT) models in large data sets may come with impractical computational burdens when general-purpose Markov chain Monte Carlo (MCMC) samplers are employed. Variational Bayes (VB)—a method for approximating the posterior distribution—poses a potential remedy. Stan’s general-purpose VB algorithms have drastically improved the accessibility of VB methods for a wide psychometric audience. Using marginal maximum likelihood (MML) and MCMC as benchmarks, the present simulation study investigates the utility of Stan’s built-in VB function for estimating multidimensional IRT models with between-item dimensionality. VB yielded a marked speed-up in comparison to MCMC, but did not generally outperform MML in terms of run time. VB estimates were trustworthy only for item difficulties, while bias in item discriminations depended on the model’s dimensionality. Under realistic conditions of non-zero correlations between dimensions, VB correlation estimates were subject to severe bias. The practical relevance of performance differences is illustrated with data from PISA 2018. We conclude that in its current form, Stan’s built-in VB algorithm does not pose a viable alternative for estimating multidimensional IRT models.
Highlights
Multidimensional item response theory (IRT) models are the method of choice for analyzing data from cognitive tests assessing multiple competencies
We focus on multidimensional IRT models with between-item dimensionality because we believe (1)
Note that these comparisons need to be interpreted with caution, as run times of Markov chain Monte Carlo (MCMC) and maximum likelihood (MML) are heavily dependent on the software as well as the number of iterations and nodes employed, respectively
Summary
Multidimensional item response theory (IRT) models are the method of choice for analyzing data from cognitive tests assessing multiple competencies (e.g., science, mathematical literacy, and reading). Markov chain Monte Carlo (MCMC) samplers that were developed for specific IRT models, such as the Gibbs sampler by [7] for the Rasch model, or they can use general-purpose software for Bayesian estimation such as JAGS [8] or Stan [9]. In our experience, applied researchers typically employ the latter software packages because they provide them with a high flexibility in model specification. This allows them to take the specifics of their data into account without the need to develop their own customized samplers (for which they may not have the time). The last point may be a disadvantage, at least when there is little research on the performance of these new techniques as implemented in the general-purpose software
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
