Classical Measurement Model Research Articles

Development and Validation of Psychometric Properties of the 10 IB Learner Profile Instrument (10IBLP-I): A Combination of the Rasch and Classical Measurement Model.

Background: The International Baccalaureate Middle Years Programme (IBMYP) aims to produce a holistic transformation with creative and critically minded students. However, very little attention has been paid to the development of an instrument to measure the IB learner profile with good psychometric properties. Purpose: This study aims to develop an instrument with good psychometric properties, based on the Rasch measurement model and confirmatory factor analysis. Methods: The study consists of two phases of pilot and field studies involving 597 year four students from IBWS MOE. Results: The findings from the Rasch measurement model analysis have shown that 54 items meet the criteria of the item fit, unidimensionality, and reliability index. Meanwhile, confirmatory factor analysis found that 44 items have shown a valid item fit index. Conclusions: The combination of both analyses has shown the strength of 10IBLP-I psychometric properties that cover the aspects of validity and reliability. The findings also provide an implication to the theory, with empirical evidence that the IB learner profile consists of 10 constructs. Besides, the evidenced 10IBLP-I comprises good psychometric properties, which can be used to measure the level of IB learner profile among IBWS MOE students to assess the effectiveness of the implementation of IBMYP in Malaysia.

Development of the student course cognitive engagement instrument (SCCEI) for college engineering courses

BackgroundEvidence shows that students who are actively engaged with learning materials demonstrate greater learning gains than those who are passively engaged. Indeed, cognitive engagement is often cited as a critical component of an educational experience. However, understanding how and in what ways cognitive engagement occurs remains a challenge for engineering educators. In particular, there exists a need to measure and evaluate engagement in ways that provide information for instructors to deploy concrete, actionable steps to foster students’ cognitive engagement. The present study reports the development and gathering of validation evidence for a quantitative instrument to measure students’ in-class cognitive engagement. The instrument was informed by Wylie and Chi’s ICAP (Interactive Constructive Active Passive) model of active learning, as well as contextual concerns within engineering courses.ResultsThe process followed the classical measurement model of scale development. We provide a detailed overview of the item development and scale validation processes, focusing on the creation of individual subscales to measure different modes of cognition within learning contexts. Multiple rounds of testing the student course cognitive engagement instrument (SCCEI) in college engineering courses provided evidence of validity. This indicated the reliable measurement of student cognitive engagement in the context of notetaking, processing material, and interacting with peers in the classroom. Results suggest differentiating modes of cognitive engagement is indeed applicable when considering students’ in-class notetaking and processing of material.ConclusionsFindings point towards the need for additional engagement scales that expand the instrument’s ability to distinguish between particular activities within a mode of engagement as defined by ICAP. The present study contributes to the growing body of literature on cognitive engagement of engineering students. Results address the development of measurement tools with evidence of validity for use in STEM education.

АНАЛІЗ ТЕСТОВИХ ЗАВДАНЬ ДЛЯ ОЦІНЮВАННЯ ЗАГАЛЬНОПЕДАГОГІЧНОЇ КОМПЕТЕНТНОСТІ ВЧИТЕЛІВ В УМОВАХ ПІСЛЯДИПЛОМНОЇ ПЕДАГОГІЧНОЇ ОСВІТИ ЗАСОБАМИ IRT

The article is devoted to the analysis of test tasks designed to assess the general pedagogical competence of teachers in postgraduate pedagogical education, and their main characteristics by means of modern testing theory. Designed tests for the start and final assessment are one of the effective, objective and reliable tools for assessing the level of general pedagogical competence of teachers in the context of postgraduate pedagogical education. During the approbation, significant statistical material was accumulated, the mathematical and statistical processing of which by means of modern IRT testing theory made it possible to determine estimates of latent parameters of subjects and parameters of test tasks by applying classical measurement models. Due to the invariance property of the IRT mathematical apparatus, the performed calculations ensure the objectivity of the assessment of the level of training of each student, which does not depend on the difficulty of the test tasks. This made it possible to correctly compare the test results of teachers who performed test tasks of different difficulty. The estimates of the difficulty of the test items obtained by the algorithm are also invariant with respect to the level of pedagogical training of applicants for education in the tested group. The characteristic curves of the difficulty level of the test tasks and the level of training of the test participants were constructed using the Rush model. The analysis of their mutual arrangement made it possible to identify the ways of further improvement of the test, create parallel tests, form a system of tasks that are most effective for assessing the level of pedagogical training of each applicant for education. When designing the test, it should be borne in mind that the proposed tasks should be different in content, form and complexity, then the designed test will carry more information about the subjects and will be suitable for assessing the level of general pedagogical competence of teachers in postgraduate pedagogical education. The testing of teachers of different specialties made it possible to check the quality of the developed test tasks and to establish the level of general pedagogical competence of the tested teachers in the conditions of postgraduate pedagogical education. Calibrated test items were entered in the bank of tasks on pedagogy and were used in the educational process.

Interactive Multiple Model Target Tracking Based on Seventh-Degree Spherical Simplex-Radial Cubature Information Filter

In this paper, we propose a new IMM (Interactive Multiple Model) algorithm called seventh degree cubature interactive multiple models IMM applied to manoeuvring Target tracking. Instead of using classical measurement model, it is proposed to consider full Doppler measurement signal as a new nonlinear observation, being highly nonlinear, and by assuming multiple and sequential measurement, information filter instead of the error covariance Kalman filter derivation is then valorized. Aiming at improving the accuracy and quick response of the filter in nonlinear manoeuvring target tracking problems, the Interacting Multiple Models 7th degree Cubature Information Filter (IMM7thCIF) is then implemented. It evaluates the information vector and information matrix rather than state vector and covariance with higher degrees than proposed in the literature, which can reduce the error of nonlinear filtering algorithm, specifically when highly nonlinear measurement are faced such as for Doppler signal. Simulation results show that the proposed filter exhibits fast and more accurate estimation and faster switching when disposing different manoeuvre models; it performs better than the IMM5th degree CKF, IMM3th degree CKF and IMMUKF on tracking accuracy.

Geometric structure of the joint N‐voM distribution manifold and its applications to sensor networks

In this paper, the geometric structure for normal distribution manifold, von Mises distribution manifold and their joint distribution manifold are firstly given by the metric, curvature, and divergence, respectively. Furthermore, the active detection with sensor networks is presented by a classical measurement model based on metric manifold, and the information resolution is presented for the range and angle measurements sensor networks. The preliminary analysis results introduced in this paper indicate that our approach is able to offer consistent and more comprehensive means to understand and solve sensor network problems containing sensors management and target detection, which are not easy to be handled by conventional analysis methods.

Information Submanifold Based on SPD Matrices and Its Applications to Sensor Networks

In this paper, firstly, manifoldPD(n)consisting of alln×nsymmetric positive-definite matrices is introduced based on matrix information geometry; Secondly, the geometrical structures of information submanifold ofPD(n)are presented including metric, geodesic and geodesic distance; Thirdly, the information resolution with sensor networks is presented by three classical measurement models based on information submanifold; Finally, the bearing-only tracking by single sensor is introduced by the Fisher information matrix. The preliminary analysis results introduced in this paper indicate that information submanifold is able to offer consistent and more comprehensive means to understand and solve sensor network problems for targets resolution and tracking, which are not easily handled by some conventional analysis methods.

Measurement Error Corrected Sodium and Potassium Intake Estimation Using 24-Hour Urinary Excretion

Epidemiological studies of the association of sodium and potassium intake with cardiovascular disease risk have almost exclusively relied on self-reported dietary data. Here, 24-hour urinary excretion assessments are used to correct the dietary self-report data for measurement error under the assumption that 24-hour urine recovery provides a biomarker that differs from usual intake according to a classical measurement model. Under this assumption, dietary self-reports underestimate sodium by 0% to 15%, overestimate potassium by 8% to 15%, and underestimate sodium/potassium ratio by ≈20% using food frequency questionnaires, 4-day food records, or three 24-hour dietary recalls in Women's Health Initiative studies. Calibration equations are developed by linear regression of log-transformed 24-hour urine assessments on corresponding log-transformed self-report assessments and several study subject characteristics. For each self-report method, the calibration equations turned out to depend on race and age and strongly on body mass index. After adjustment for temporal variation, calibration equations using food records or recalls explained 45% to 50% of the variation in (log-transformed) 24-hour urine assessments for sodium, 60% to 70% of the variation for potassium, and 55% to 60% of the variation for sodium/potassium ratio. These equations may be suitable for use in epidemiological disease association studies among postmenopausal women. The corresponding signals from food frequency questionnaire data were weak, but calibration equations for the ratios of sodium and potassium/total energy explained ≈35%, 50%, and 45% of log-biomarker variation for sodium, potassium, and their ratio, respectively, after the adjustment for temporal biomarker variation and may be suitable for cautious use in epidemiological studies. Clinical Trial Registration- URL: www.clinicaltrials.gov. Unique identifier: NCT00000611.

Od alfe do omege i natrag

Koeficijent α, poznat i kao koeficijent interne konzistencije, jedna je od najčešće korišćenih i istovremeno najviše zloupotrebljavanih procena pouzdanosti kompozitnih mernih instrumenata ili testova. Za tumačenje rezultata dobijenih primenom nekog psihološkog testa najvažnija zloupotreba koeficijenta α je pogrešno zaključivanje o njegovoj zadovoljavajućoj pouzdanosti i homogenosti ili jednodimenzionalnosti. Jedan izvor ovakve vrste zloupotrebe proizilazi iz same definicije koeficijenta α kao procene pouzdanosti tipa interne konzistencije. Drugi razlog ovakvoj zloupotrebi nalazi se u nedovoljnom poznavanju razlika u poimanju i definicijama pouzdanosti, homogenosti i valjanosti između klasičnog modela merenja i njegove usavršene verzije, faktorskoanalitičkog modela merenja. Zbog toga su već duže u opticaju i alternativni predlozi, kao što je koeficijent ω koji predstavlja bolju procenu pouzdanosti, homogenosti i valjanosti pod Spearmanovim konfirmatornim jednofaktorskim modelom. Na osnovu podataka nastalih primenom jednog instrumenta putem elementarne algebre, demonstrirane su i upoređene procene koeficijenata α i ω. Rezultati ovih, kao i drugih analiza, govore da je povoljna procena α neophodan, ali ne i dovoljan razlog za zaključak o pouzdanosti i homogenosti testa. Da bi se o tome moglo utemeljeno zaključivati, potrebne su dodatne procene kao što je ω, zajedno sa procenama saglasnosti modela sa podacima, pogotovo u onim, za ovakvu vrstu instrumenata, čestim slučajevima koji zahtevaju proveru jednofaktorskog modela merenja.

Exploring Psychometric Models to Enhance Standardized Patient Quality Assurance

To explore methods to assess standardized patient (SP) performance over time so as to sustain objective performance in a high-stakes clinical skills examination and to inform quality assurance. The authors selected data from the United States Medical Licensing Examination-Step 2 Clinical Skills to assess the relative usefulness of the classical measurement and common factor models in determining the difficulty and discrimination of SP-medical case pairs (SP-cases) on communication scores over time. The common factor model is an alternative to the classical measurement model and can be used to calibrate SP-case parameters. The sample comprised 88 SP-case combinations in test administrations throughout the year 2010. The authors constructed four time segments from scoring cohorts; computed, for each method, difficulty and discrimination parameters for each SP-case within each time segment; and then compared the efficacy of each. They also compared qualitative SP-case performance standards established through video monitoring to the common factor model for relative usefulness in identifying SP-case outliers. SP-case difficulty parameters produced by the classical measurement and common factor models were similarly useful for SP performance evaluation over time. The SP-case discrimination parameters produced by the common factor model appeared to capture more variation in performance. Although either method is equally useful for assessing SP-case difficulty, the common factor model is more sensitive to fluctuations in SP-case discrimination and could be an additionally useful source of information to identify outliers and to direct quality assurance resources for routine SP evaluation.

A Rasch Perspective

Measurement error is a common theme in classical measurement models used in testing and assessment. In classical measurement models, the definition of measurement error and the subsequent reliability coefficients differ on the basis of the test administration design. Internal consistency reliability specifies error due primarily to poor item sampling. Rater reliability indicates error due to inconsistency among raters. For estimates of test-retest reliability, error is attributed mainly to changes over time. In alternate-forms reliability, error is assumed to be due largely to variation between samples of items on test forms. Rasch models can also compute reliability estimates of scores under different test situations. The authors therefore present the Rasch perspective on calculating reliability (measurement error) and present Rasch measurement model programs to compute the various reliability estimates.

Maximum Likelihood Estimation of Generalized Linear Models with Covariate Measurement Error

Generalized linear models with covariate measurement error can be estimated by maximum likelihood using gllamm, a program that fits a large class of multilevel latent variable models (Rabe-Hesketh, Skrondal, and Pickles 2004). The program uses adaptive quadrature to evaluate the log likelihood, producing more reliable results than many other methods (Rabe-Hesketh, Skrondal, and Pickles 2002). For a single covariate measured with error (assuming a classical measurement model), we describe a “wrapper” command, cme, that calls gllamm to estimate the model. The wrapper makes life easy for the user by accepting a simple syntax and data structure and producing extended and easily interpretable output. The commands for preparing the data and running gllamm can also be obtained from cme. We first discuss the case where several measurements are available and subsequently consider estimation when the measurement error variance is instead assumed known. The latter approach is useful for sensitivity analysis assessing the impact of assuming perfectly measured covariates in generalized linear models. An advantage of using gllamm directly is that the classical covariate measurement error model can be extended in various ways. For instance, we can use nonparametric maximum likelihood estimation (NPMLE) to relax the normality assumption for the true covariate. We can also specify a congeneric measurement model which relaxes the assumption that the measurements for a unit are exchangeable replicates by allowing for different measurement scales and error variances.

Maximum Likelihood Estimation of Generalized Linear Models with Covariate Measurement Error

Generalized linear models with covariate measurement error can be estimated by maximum likelihood using gllamm, a program that fits a large class of multilevel latent variable models (Rabe-Hesketh, Skrondal, and Pickles 2004). The program uses adaptive quadrature to evaluate the log likelihood, produc- ing more reliable results than many other methods (Rabe-Hesketh, Skrondal, and Pickles 2002). For a single covariate measured with error (assuming a classical measurement model), we describe a "wrapper" command, cme, that calls gllamm to estimate the model. The wrapper makes life easy for the user by accepting a simple syntax and data structure and producing extended and easily interpretable output. The commands for preparing the data and running gllamm can also be obtained from cme. We first discuss the case where several measurements are avail- able and subsequently consider estimation when the measurement error variance is instead assumed known. The latter approach is useful for sensitivity analy- sis assessing the impact of assuming perfectly measured covariates in generalized linear models. An advantage of using gllamm directly is that the classical co- variate measurement error model can be extended in various ways. For instance, we can use nonparametric maximum likelihood estimation (NPMLE) to relax the normality assumption for the true covariate. We can also specify a congeneric measurement model which relaxes the assumption that the measurements for a unit are exchangeable replicates by allowing for different measurement scales and error variances.

A Risk Set Calibration Method for Failure Time Regression by Using a Covariate Reliability Sample

Summary Regression parameter estimation in the Cox failure time model is considered when regression variables are subject to measurement error. Assuming that repeat regression vector measurements adhere to a classical measurement model, we can consider an ordinary regression calibration approach in which the unobserved covariates are replaced by an estimate of their conditional expectation given available covariate measurements. However, since the rate of withdrawal from the risk set across the time axis, due to failure or censoring, will typically depend on covariates, we may improve the regression parameter estimator by recalibrating within each risk set. The asymptotic and small sample properties of such a risk set regression calibration estimator are studied. A simple estimator based on a least squares calibration in each risk set appears able to eliminate much of the bias that attends the ordinary regression calibration estimator under extreme measurement error circumstances. Corresponding asymptotic distribution theory is developed, small sample properties are studied using computer simulations and an illustration is provided.

RELIABILITY IS NOT VALIDITY AND VALIDITY IS NOT RELIABILITY

Interrater correlations do provide an index of reliability of job performance ratings. We show that the arguments presented by Murphy and DeShon (2000) lead to the radical conclusion that traditional measurement models–both classical theory and generalizability theory models–can be used neither with job performance ratings nor with other measures used in I‐O and other areas of psychology and the social sciences. We show that this untenable conclusion is based on confusion of validity issues and questions with reliability issues and questions. It is also based on the incorrect belief that classical measurement models are capable of addressing only random response measurement error and cannot address other forms of measurement error. We also show that the solution Murphy and DeShon offer to the problem of measurement error in ratings, as they define this problem, cannot work. Properly understood, the position taken by Murphy and DeShon leaves us with the nihilistic conclusion that no appropriate measurement models are possible in psychological research, thus making meaningful research impossible.

Confirmatory Measurement Model Comparisons Using Latent Means.

Confirmatory factor analysis (CFA) is often used to verify measurement models derived from classical test theory: the parallel, tau-equivalent, and congeneric test models. In this application, CFA is traditionally applied to the observed covariance or correlation matrix, ignoring the observed mean structure. But CFA is easily extended to allow nonzero observed and latent means. The use of CFA with nonzero latent means in testing six measurement models derived from classical test theory is discussed. Three of these models have not been addressed previously in the context of CFA. The implications of the six models for observed mean and covariance structures are fully described. Three examples of the use of CFA in testing these models are presented. Some advantages and limitations in using CFA with nonzero latent means to verify classical measurement models are discussed.

On Estimating the Reliability of Composite Indexes Through Factor Analysis

The rationale for Heise and Bohrnstedt's index of reliability, omega, is restated in terms of the models of classical measurement theory and factor analysis. Although omega is the best available internal-consistency estimate of reliability, its value is generally lower than a composite scale's true reliability in the classical measurement model. Under quite foreseeable circumstances, it may be appreciably lower. Some implications for forming indexes through factor analysis are discussed.