Weighted Likelihood Research Articles

Constructing a Robust Score Scale from IRT Scores with Informed Boundaries

AbstractIn operational testing, item response theory (IRT) models for dichotomous responses are popular for measuring a single latent construct , such as cognitive ability in a content domain. Estimates of , also called IRT scores or , can be computed using estimators based on the likelihood function, such as maximum likelihood (ML), weighted likelihood (WL), maximum a posteriori (MAP), and expected a posteriori (EAP). Although the parameter space of is theoretically unrestricted, the range of finite is constrained by the estimator and test form properties, which is important to consider but often overlooked when developing a score scale for reporting purposes. Irrespective of the estimator or test forms at hand, a common practice is to fix arbitrary points symmetric about zero (e.g., −4 and 4) as anchors for deriving a score transformation, possibly resulting in unintended gaps or truncations at the extremes. Therefore, a systematic framework is proposed for using IRT scores to construct a robust score scale with informed boundaries that are logical and consistent across test forms.

Analysis of an outcome-dependent enriched sample: hypothesis tests

An outcome-dependent sample is generated by a stratified survey design where the stratification depends on the outcome. It is also known as a case–control sample in epidemiological studies and a choice-based sample in econometrical studies. An outcome-dependent enriched sample (ODE) results from combining an outcome-dependent sample with an independently collected random sample. Consider the situation where the conditional probability of a categorical outcome given its covariates follows an explicit model with an unknown parameter whereas the marginal probability of the outcome and its covariates are left unspecified. Profile-likelihood (PL) and weighted-likelihood (WL) methods have been employed to estimate the model parameter from an ODE sample. This article develops the PL- and WL-based families of tests on the model parameter from an ODE sample. Asymptotic properties of their test statistics are derived. The PL likelihood-ratio, Wald and score tests are shown to obey classical inference, i.e. their test statistics are asymptotically equivalent and Chi-squared distributed. In contrast, the WL likelihood-ratio statistic asymptotically has a weighted Chi-squared distribution and is not equivalent to the WL Wald and score statistics. Our theoretical derivation and simulation show that tests based on these new statistics carry nominal type I error and good power. Advantages of ODE sampling together with the implementation of the PL and WL methods are demonstrated in an illustrative example.

Accuracy of Asymptotic Standard Errors of the Maximum and Weighted Likelihood Estimators of Proficiency Levels With Short Tests

The maximum likelihood (ML) and the weighted likelihood (WL) estimators are commonly used to obtain proficiency level estimates with pre-calibrated item parameters. Both estimators have the same asymptotic standard error (ASE) that can be easily derived from the expected information function of the test. However, the accuracy of this asymptotic formula is unclear with short tests when only a few items are administered. The purpose of this paper is to compare the ASE of these estimators with their exact values, evaluated at the proficiency-level estimates. The exact standard error ( SE) is computed by generating the full exact sample distribution of the estimators, so its practical feasibility is limited to small tests (except under the Rasch model). A simulation study was conducted to compare the ASE and the exact SE of the ML and WL estimators, with the “true” SE (i.e., computed as the exact SE with the true proficiency levels). It is concluded that with small tests, the exact SEs are less biased and return smaller root mean square error values than the asymptotic SEs, while as expected, the two estimators return similar results with longer tests.

On the Relationships Between Jeffreys Modal and Weighted Likelihood Estimation of Ability Under Logistic IRT Models

This paper focuses on two estimators of ability with logistic item response theory models: the Bayesian modal (BM) estimator and the weighted likelihood (WL) estimator. For the BM estimator, Jeffreys’ prior distribution is considered, and the corresponding estimator is referred to as the Jeffreys modal (JM) estimator. It is established that under the three-parameter logistic model, the JM estimator returns larger estimates than the WL estimator. Several implications of this result are outlined.