Conditional Standard Error Of Measurement Research Articles

Rotation Local Solutions in Multidimensional Item Response Theory Models

We conducted an extensive Monte Carlo study of factor-rotation local solutions (LS) in multidimensional, two-parameter logistic (M2PL) item response models. In this study, we simulated more than 19,200 data sets that were drawn from 96 model conditions and performed more than 7.6 million rotations to examine the influence of (a) slope parameter sizes, (b) number of indicators per factor (trait), (c) probabilities of cross-loadings, (d) factor correlation sizes, (e) model approximation error, and (f) sample sizes on the local solution rates of the oblimin and (oblique) geomin rotation algorithms. To accommodate these design variables, we extended the standard M2PL model to include correlated major factors and uncorrelated minor factors (to represent model error). Our results showed that both rotation methods converged to LS under some conditions with geomin producing the highest local solution rates across many models. Our results also showed that, for identical item response patterns, rotation LS can produce different latent trait estimates with different levels of measurement precision (as indexed by the conditional standard error of measurement). Follow-up analyses revealed that when rotation algorithms converged to multiple solutions, quantitative indices of structural fit, such as numerical measures of simple structure, will often misidentify the rotation that is closest in mean-squared error to the factor pattern (or item-slope pattern) of the data-generating model.

Student achievement, school quality, and an error-prone family background measure: exploring the sensitivity of the Heyneman-Loxley effect in Southern and Eastern Africa

BackgroundWe examine the sensitivity of the Heyneman-Loxley Effect to the influence of an error-prone family background measure in 15 education systems from Southern and Eastern Africa. Our aim is to revisit a claim by Abby Riddell from the November 1989 issue of the Comparative Education Review concerning the reliability of family background measures and the estimation of the Heyneman-Loxley Effect. Three questions guide our study: does national income have an association with the reliability of a family background measure, is the association between a family background measure and student achievement sensitive to measurement error, and is the association between national income and the school effect sensitive to measurement error?MethodsOur analysis relies on the SACMEQ III data archive and, most importantly, a known error-prone family background measure (i.e., socioeconomic status index) and its corresponding measurement error (i.e., conditional standard error of measurement). For each SACMEQ III education system, we calculate the reliability of the socioeconomic status index and examine its association with national income. We use a Bayesian multilevel regression model to estimate naive and correction parameters representing the association between the socioeconomic status index and student achievement. Finally, we explore the associations between national income and the naive and correction estimates for the school effect across SACMEQ III education systems.ResultsWe observe three results. First, the association between national income and the reliability of the socioeconomic status index appears negative among SACMEQ III education systems (albeit questionable due to the small n-size and influential outliers). Second, the association between the socioeconomic status index and student achievement is sensitive to measurement error across content areas and SACMEQ III education systems. Third and finally, the association between national income and the school effect is insensitive to measurement error across content areas and SACMEQ III education systems.ConclusionsThroughout our study, we discuss measurement error, its consequences, and why the correction of error-prone family background measures is important. We highlight the need for auxiliary information for measurement error correction (e.g., reliability ratio, conditional standard error of measurement). Lastly, in addition to encouraging the correction of error-prone family background measures when attempting to replicate the Heyneman-Loxley Effect, we invite further research on improving the reliability and comparability of family background measures.

An Evaluation of the Forecasting Accuracy of Two Trend Estimation Methods from Two Computer Adaptive Tests

This study evaluated the forecasting accuracy of trend estimation methods applied to time-series data from computer adaptive tests (CATs). Data were collected roughly once a month over the course of a school year. We evaluated the forecasting accuracy of two regression-based growth estimation methods (ordinary least squares and Theil-Sen). The precision, or accuracy, of predictions were heavily influenced by how far performance was forecasted into the future (1, 3, or 5 months) and the number of observations available to estimate growth (3, 4, or 5). When performance was forecasted further and growth estimates were based upon fewer observations, predictions were off by as much as two times the average conditional standard error of measurement for a grade level. No combinations of data collections schedule, forecasting length, or estimation method led to consistent bias in predictions. Educators should be cautious when using trend-line decision rules to predict future performance. Suggestions for alternate decision rules to explore with CATs are offered.

ITEM BANK DEVELOPMENT AND VALIDATION FOR ABILITY ESTIMATION: COMPUTER ADAPTIVE TESTING APPLICATION FOR POSTGRADUATE CERTIFICATION/DIPLOMA IN EDUCATION (MATHEMATICS)

Ponte Academic JournalOct 2022, Volume 78, Issue 10 ITEM BANK DEVELOPMENT AND VALIDATION FOR ABILITY ESTIMATION: COMPUTER ADAPTIVE TESTING APPLICATION FOR POSTGRADUATE CERTIFICATION/DIPLOMA IN EDUCATION (MATHEMATICS)Author(s): Jumoke I. Oladele ,Mdutshekelwa Ndlovu, Erica D. Spangenberg, Virendra RamdhanyJ. Ponte - Oct 2022 - Volume 78 - Issue 10 doi: 10.21506/j.ponte.2022.10.8 Abstract:The foundation of a good psychological measurement, such as ability estimation, is to develop an optimal item bank for educational assessments. This research aimed to develop an optimal item bank for the further teacher training programme in Mathematics while detailing all the procedures, including planning, content analysis, and test blueprint going by Bloom's taxonomy of educational objectives. The pre-experimental research design using the one-shot case study type was adopted for the study and complemented by the feature data generation FDG technique. The technique was deployed through a post-hoc simulation using item parameter estimates with real data collected from 35 participants with 68 multiple-choice items, from which 200 items with 535 participants were generated. The developed item bank was validated for ability estimation using the item response theory model with three logistical parameters: item difficulty level, item discriminating power, and pseudo-guessing. The responses to the test resulted in the collection and analysis of dichotomous scored tests for answering the generated research questions. The item parameters and ability estimation were analyzed using the Xcalibre 4.2 programme. Ninety per cent of the items met the criteria of a good item, including having a good difficulty level and functioning properly between the high and low ability participants with the discrimination and guessing parameters. A relatively low Conditional Standard Error of Measurement was recorded as the indices for ascertaining the precision of ability estimation using the developed test based on which recommendations were made. The carefully outlined procedures also ensure replicability within other programmes. Download full text:Check if you have access through your login credentials or your institution Username Password

Conditional Standard Error of Measurement: Classical Test Theory, Generalizability Theory and Many-Facet Rasch Measurement with Applications to Writing Assessment.

Conditional Standard Error of Measurement: Classical Test Theory, Generalizability Theory and Many-Facet Rasch Measurement with Applications to Writing Assessment.

Addressing the Large Standard Error of Traditional CBM-R: Estimating the Conditional Standard Error of a Model-Based Estimate of CBM-R

Curriculum-based measurement of oral reading fluency (CBM-R) is widely used across the country as a quick measure of reading proficiency that also serves as a good predictor of comprehension and overall reading achievement, but it has several practical and technical inadequacies, including a large standard error of measurement (SEM). Reducing the SEM of CBM-R scores has positive implications for educators using these measures to screen or monitor student growth. The purpose of this study was to compare the SEM of traditional CBM-R words correct per minute (WCPM) fluency scores and the conditional SEM (CSEM) of model-based WCPM estimates, particularly for students with or at risk of poor reading outcomes. We found (a) the average CSEM for the model-based WCPM estimates was substantially smaller than the reported SEMs of traditional CBM-R systems, especially for scores at/below the 25th percentile, and (b) a large proportion (84%) of sample scores, and an even larger proportion of scores at/below the 25th percentile (about 99%) had a smaller CSEM than the reported SEMs of traditional CBM-R systems.

수학적 창의성 태도 검사에서 수학영재와 일반학생의 다집단 일반화가능도 분석

본 연구의 목적은 측정학적 이론을 바탕으로 수학영재 집단과 일반학생 집단에서 수학적 창의성 태도 검사 점수에 영향을 미치는 오차 요인들의 상대적인 영향력과 적정 수준의 신뢰도에 도달하는 효율적인 측정 조건을 탐색하는 데 있다. 이를 위해 109명의 중등 수학영재와 125명의 일반 중학생을 대상으로 실시한 수학적 창의성 태도 검사 결과에 다변량 일반화가능도 분석을 수행하였다. 주요 분석 결과는 다음과 같다. 첫째, 수학적 창의성 태도 검사는 신뢰도를 기준으로는 일반학생 집단에서, 조건부 측정 오차를 기준으로는 수학영재 집단에서 좀 더 적합한 것으로 나타났지만, 두 집단에서 모두 시행할 수 있는 신뢰로운 측정도구인 것으로 나타났다. 둘째, 수학적 창의성 태도 검사에서 신뢰도를 높이기 위해서는 수학영재 집단의 경우 수렴적 태도를 높게. 반면에 일반학생 집단의 경우 발산적 태도와 문제해결 태도를 높게 반영하여야 하는 것으로 나타났다. 셋째, 수학영재 집단의 경우 발산적 태도, 문제해결 태도, 수렴적 태도의 다차원적인 요소를 반영하는 경우, 그리고 일반학생 집단의 경우 발산적 태도만으로도 적정 수준의 신뢰도에 도달하는 것으로 나타났다. 마지막으로 이러한 연구결과들을 바탕으로 수학적 창의성 태도 검사 활용 방안 및 향후 연구 방향을 제시하였다. The purpose of this study was to investigate the relative influence of multiple error sources and to find optimal measurement conditions that obtain a desired level of reliability of a creative attitude test in mathematical creativity. This study analyzed the scores of the Creative Attitude Scale-Korea allowed to access publicly of 125 general students and 109 mathematically gifted students by performing a multivariate generalizability analysis. The main results were as follows. First, based on reliability, the Creative Attitude Scale-Korea was measured less precisely for mathematically gifted students. On the contrary, based on the conditional standard error of measurement, it was measured less precisely for general students. However, the Creative Attitude Scale-Korea showed strong reliability in both groups. Second, the optimal weights should adjust to .3, .3, .4 in mathematically gifted students and .4, .4, .2 in general students with three scoring components of divergent attitude, problem solving attitude, and convergent attitude based on the maximum reliability. Third, to approach desirable reliability, it is possible to use one component of divergent attitude in general students but three components of divergent attitude, problem solving attitude, and convergent attitude in mathematically gifted students. Finally this study proposed application plans for the Creative Attitude Scale-Korea and future directions of research.

Development and validation of the Granada Burnout Questionnaire in Spanish police

The goal of the present study is to develop a questionnaire, with proper psychometric properties and current norms, to evaluate the burnout syndrome in Spain. The operative definition of burnout proposed by Maslach and Jackson is used to define three dimensions (Emotional exhaustion, Depersonalization and Personal accomplishment). A total of 2,403 national Spanish police participated. Evidence of construct validity was checked through cross validation (showing a good fit of the three factors model to the data). Using the MBI, NEO-FII and CECAD evidence of convergent validity and criteria validity were developed (showing that the relations are similar to the ones that appear in other research). The discrimination, mean, standard deviation, and typical error of the average of the items composing the various dimensions were analyzed. Both the Cronbach's alpha coefficient and the conditional standard error of measurement (CSEM) were calculated for each of the dimensions. The results showed good internal consistency (all α values > .85). Finally, the questionnaire was scaled using T scores. The psychometrical properties reported here support the use of this new questionnaire for the burnout evaluation in Spanish police.

A CRITERION TO EVALUATE THE INDIVIDUAL RAW‐TO‐SCALE EQUATING CONVERSIONS

ABSTRACTIn this study we investigated when an equating conversion line is problematic in terms of gaps and clumps. We suggest using the conditional standard error of measurement (CSEM) to measure the scale scores that are inappropriate in the overall raw‐to‐scale transformation.

The Reliability and Precision of Total Scores and IRT Estimates as a Function of Polytomous IRT Parameters and Latent Trait Distribution

A classic topic in the fields of psychometrics and measurement has been the impact of the number of scale categories on test score reliability. This study builds on previous research by further articulating the relationship between item response theory (IRT) and classical test theory (CTT). Equations are presented for comparing the reliability and precision of scores within the CTT and IRT frameworks. This study presented new results pertaining to the relative precision (i.e., the test score conditional standard error of measurement for a given trait value) of CTT and IRT, and the new results shed light on the conditions where total scores and IRT estimates are more or less precisely measured. The relative reliability of CTT and IRT scores is examined as a function of item characteristics (e.g., locations, category thresholds, and discriminations) and subject characteristics (e.g., the skewness and kurtosis of the latent distribution). CTT total scores were more reliable when the latent distribution was mismatched with category thresholds, but the discrepancy between CTT and IRT declined as the number of scale categories increased. This article also considered the appropriateness of linear approximations of polytomous items and presented circumstances where linear approximations are viable. A linear approximation may be appropriate for items with two response options depending on the item discrimination and the match between the item location and latent distribution. However, linear approximations are biased whenever items are located in the tails of the latent distribution and the bias is larger for more discriminating items.

High School Exit Exams and Mismeasurement

Test score validity takes center stage in the debate over the use of high school exit exams. Scant literature addresses the amount of conditional standard error of measurement (CSEM) present in individual student results on high school exit exams. The purpose of this study is to fill a void in the literature and add a national review of the CSEM, including data on the amount of CSEM present in high school exit exams results. Individual student results from each of the 23 exit exams contained a CSEM ranging from 3.29 to 39 scale-score points. Nearly one-fourth of the state education agencies did not report the CSEM for the individual student results.

COMPARISON OF MULTISTAGE TESTS WITH COMPUTERIZED ADAPTIVE AND PAPER‐AND‐PENCIL TESTS

ABSTRACTTraditionally, the fixed‐length linear paper‐and‐pencil (P&P) mode of administration has been the standard method of test delivery. With the advancement of technology, however, the popularity of administering tests using adaptive methods like computerized adaptive testing (CAT) and multistage testing (MST) has grown in the field of measurement in both theory and practice. In practice, several standardized tests have sections that include only set‐based items. To date, there is no study in the literature that compares these testing procedures when a test is completely set‐based under various item response theory (IRT) models. This study investigates the measurement precision of MST compared to CAT and compared to P&P tests for the one‐, two‐, and three‐parameter logistic (1‐, 2‐, and 3PL) models when the test is completely set‐based. Results showed that MST performed better for the 2‐ and 3PL models than an equivalent‐length P&P test in terms of reliability and conditional standard error of measurement. In addition, findings showed that MST performed better for the 1‐ and 2PL models than for an equivalent‐length CAT test. For the 3PL model, MST and CAT performed about the same.

Standardized Conditional SEM: A Case for Conditional Reliability

An examinee-level (or conditional) reliability is proposed for use in both classical test theory (CTT) and item response theory (IRT). The well-known group-level reliability is shown to be the average of conditional reliabilities of examinees in a group or a population. This relationship is similar to the known relationship between the square of the conditional standard error of measurement (SEM) and the square of the group-level SEM. The proposed conditional reliability is illustrated with an empirical data set in the CTT and IRT frameworks.

Conditional Standard Errors of Measurement for Composite Scores on the Wechsler Preschool and Primary Scale of Intelligence-Third Edition

A specific recommendation of the 1999 Standards for Educational and Psychological Testing by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education is that test publishers report estimates of the conditional standard error of measurement (SEM). Procedures for calculating the conditional (score-level) SEM based on raw scores are well documented; however, few procedures have been developed for estimating the conditional SEM of subtest or composite scale scores resulting from a nonlinear transformation. Item response theory provided the psychometric foundation to derive the conditional standard errors of measurement and confidence intervals for composite scores on the Wechsler Preschool and Primary Scale of Intelligence-Third Edition.

Conditional standard error of measurement and personality scale scores: an investigation of classical test theory estimates with four MMPI scales

Abstract This study is a preliminary examination of the fit of three classical test theory models of standard error of measurement to selected personality scale (MMPI) score retest data. The three models compared are the conventional standard error of measurement formula, Lord’s (1955: Lord, F. M. (1955). Estimating test reliability. Educational and Psychological Measurement, 15, 325–336) conditional standard error of measurement formula, and Keats’ (1957: Keats, J. A. (1957). Estimation of error variances of test scores. Psychometrika, 22, 29–41) conditional standard error of measurement formula. Absolute mean error differences between test and retest MMPI scale scores were calculated from boot-strapping (with replacement) a sample of MMPI scale retest scores for 271 psychiatric patients. It was found that, on average across 5000 samplings, Keats’ conditional standard error of measurement formula produced less average absolute difference between estimated and actual MMPI absolute test–retest scale score differences than did the conventionally-employed standard error of measurement formula, and less average absolute error difference than Lord’s conditional standard error of measurement. Potential implications for practitioners in establishing cut-scores for patient classification are raised.

Psychometric Properties of Scale Scores and Performance Levels for Performance Assessments Using Polytomous IRT

With a focus on performance assessments, this paper describes procedures for calculating conditional standard error of measurement (CSEM) and reliability of scale scores and classification consistency of performance levels. Scale scores that are transformations of total raw scores are the focus of these procedures, although other types of raw scores are considered as well. Polytomous IRT models provide the psychometric foundation for the procedures that are described. The procedures are applied using test data from ACT's Work Keys Writing Assessment to demonstrate their usefulness. Two polytomous IRT models were compared, as were two different procedures for calculating scores. One simulation study was done using one of the models to evaluate the accuracy of the proposed procedures. The results suggest that the procedures provide quite stable estimates and have the potential to be useful in a variety of performance assessment situations.

A Comparison of Methods of Estimating Conditional Standard Errors of Measurement for Testlet‐Based Test Scores Using Simulation Techniques

The primary purpose of this study was to investigate the appropriateness and implication of incorporating a testlet definition into the estimation of procedures of the conditional standard error of measurement (SEM) for tests composed of testlets. Another purpose was to investigate the bias in estimates of the conditional SEM when using item‐based methods instead of testlet‐based methods. Several item‐based and testlet‐based estimation methods were proposed and compared. In general, item‐based estimation methods underestimated the conditional SEM for tests composed for testlets, and the magnitude of this negative bias increased as the degree of conditional dependence among items within testlets increased. However, an item‐based method using a generalizability theory model provided good estimates of the conditional SEM under mild violation of the assumptions for measurement modeling. Under moderate or somewhat severe violation, testlet‐based methods with item response models provided good estimates.

Estimating Conditional Standard Errors of Measurement for Tests Composed of Testlets

The primary purposes of this study were to investigate both the appropriateness and the implications of incorporating a testlet definition into the estimation of the conditional standard error of measurement (SEM) for tests composed of testlets. When individual items are used as the fundamental measurement unit with tests composed of testlets, the assumptions required by measurement modeling are violated, but those assumptions are satisfied when testlet scores are used as the measurement unit. Therefore, item-based estimation methods probably introduce some magnitude of bias in the estimates of conditional SEMs for tests composed of testlets. The five conditional SEM estimation methods used in this study were classified as either item-based or testlet-based methods. In general, the item-based methods provide lower estimates of the conditional SEM along the score scale than do the testlet-based methods.

Advances in Clinical Personality Measurement: An Item Response Theory Analysis of the MMPI-2 PSY-5 Scales

Item response theory (IRT) provides valuable methods for the analysis of the psychometric properties of a psychological measure. To date, however, these methods have not been used frequently by personality assessment researchers, in part because many researchers have not been introduced to the methods and in part because most of the development of IRT has taken place in applied education assessment settings, resulting in terminology that is ability focused rather than trait focused. The purpose of this article is twofold. First, an overview of IRT is presented, highlighting the concepts of the three-parameter IRT model, item and test information, and conditional standard error of measurement. Second, the psychometric properties of the (MMPI-2) PSY-5 scales are examined to demonstrate IRT's value.

Conditional Standard Errors of Measurement for Scale Scores Using IRT

An IRT method for estimating conditional standard errors of measurement of scale scores is presented, where scale scores are nonlinear transformations of number‐correct scores. The standard errors account for measurement error that is introduced due to rounding scale scores to integers. Procedures for estimating the average conditional standard error of measurement for scale scores and reliability of scale scores are also described. An illustration of the use of the methodology is presented, and the results from the IRT method are compared to the results from a previously developed method that is based on strong true‐score theory.