Messick's Framework Research Articles

An Evaluation of the Psychometric Properties of the Behavior Change Strategies for Healthy Eating Scale.

The Behavior Change Strategies for Healthy Eating Scale (BCSHES) has potentially broad implications for use by researchers, practitioners, and educators across various medical and allied health professions. To date, however, the psychometric properties of the BCSHES have not been rigorously evaluated, potentially limiting its use. The purpose of this study was to thoroughly evaluate the psychometric properties of the BCSHES. The Rasch Rating Scale Model was used to analyze BCSHES data on a sample of 191 graduate/professional program students. Messick's framework for construct validity was used to discern validity evidence, which noted the BCSHES possessed very strong psychometric properties and is capable of yielding valid and reliable scores. Use of the scale is encouraged, where appropriate.

Severity of Nasal Inflammatory Disease Questionnaire for Canine Idiopathic Rhinitis Control: Instrument Development and Initial Validity Evidence.

BackgroundEffective treatments are needed for idiopathic chronic rhinitis in dogs, but assessment of efficacy requires a practical, quantifiable method for assessing severity of disease.ObjectivesTo develop and perform initial validity and reliability testing of an owner‐completed questionnaire for assessing clinical signs and dog and owner quality of life (QOL) in canine chronic rhinitis.AnimalsTwenty‐two dogs with histopathologically confirmed chronic rhinitis and 72 healthy dogs.MethodsIn this prospective study, an online questionnaire was created based on literature review and feedback from veterinarians, veterinary internists with respiratory expertise, and owners of dogs with rhinitis. Owners of affected dogs completed the questionnaire twice, 1 week apart, to test reliability. Healthy dogs were assessed once. Data were analyzed using the Rasch Rating Scale Model, and results were interpreted using Messick's framework for evaluating construct validity evidence.ResultsInitial item generation resulted in 5 domains: nasal signs, paranasal signs, global rhinitis severity, and dog's and owner's QOL. A 25‐item questionnaire was developed using 5‐point Likert‐type scales. No respondent found the questionnaire difficult to complete. Strong psychometric evidence was available to support the substantive, generalizability, content, and structural aspects of construct validity. Statistical differences were found between responses for affected and control dogs for all but 2 items. These items were eliminated, resulting in the 23‐item Severity of Nasal Inflammatory Disease (SNIFLD) questionnaire.Conclusions and Clinical ImportanceThe SNIFLD questionnaire provides a mechanism for repeated assessments of disease severity in dogs with chronic rhinitis.

Surgical skills curricula in American College of Surgeons Accredited Education Institutes: An international survey

BackgroundA clear understanding of simulation-based curricula in use at American College of Surgeons Accredited Education Institutes (ACS-AEIs) is lacking. MethodsA 25-question online survey was sent to ACS-AEIs. ResultsThe response rate approached 60%. The most frequent specialties to use the ACS-AEIs are general surgery and obstetrics/gynecology (94%). Residents are the main target population for programming/training (96%). Elements of the ACS/Association of Program Directors in Surgery Surgical Skills Curriculum are used by 77% of responding ACS-AEIs. Only 49% of ACS-AEIs implement the entire curriculum and 96% have independently developed their own surgical skills curricula. “Home-grown” simulators have been designed at 71% of ACS-AEIs. Feasibility (80%), evidence of effectiveness (67%), and cost (60%) were reasons for curriculum adoption. All programs use operative assessment tools for resident performance, and 53% use Messick's unitary framework of validity. Most programs (88%) have financial support from their academic institute. Majority of ACS-AEIs had trainees evaluate their faculty instructors (90%), and the main form of such faculty evaluation was postcourse surveys (97%). ConclusionThis study provides specific information regarding simulation-based curricula at ACS-AEIs.

Validity and Feasibility Evidence of Objective Structured Clinical Examination to Assess Competencies of Pediatric Critical Care Trainees.

The purpose of this study was to provide validity and feasibility evidence for the use of an objective structured clinical examination in the assessment of pediatric critical care medicine trainees. This was a validation study. Validity evidence was based on Messick's framework. A tertiary, university-affiliated academic center. Seventeen pediatric critical care medicine fellows were recruited in 2012 and 2013 academic year. None. All subjects completed an objective structured clinical examination assessment. Seventeen trainees were assessed. Simulation scenarios were developed for content validity by pediatric critical care medicine and education experts using CanMEDS competencies. Scenarios were piloted before the study. Each scenario was evaluated by two interprofessional raters. Inter-rater agreement, measured using intraclass correlations, was 0.91 (SE = 0.09) across stations. Generalizability theory was used to evaluate internal structure and reliability. Reliability was moderate (G-coefficient = 0.67, Φ-coefficient = 0.52). The greatest source of variability was from participant by station variance (40.6%). Pearson correlation coefficients were used to evaluate the relationship of objective structured clinical examination with each traditional assessment instruments: multisource feedback, in-training evaluation report, short-answer questions, and Multidisciplinary Critical Care Knowledge Assessment Program. Performance on the objective structured clinical examination correlated with performance on the Multidisciplinary Critical Care Knowledge Assessment Program (r = 0.52; p = 0.032) and multisource feedback (r = 0.59; p = 0.017), but not with the overall performance on the in-training evaluation report (r = 0.37; p = 0.143) or short-answer questions (r = 0.08; p = 0.767). Consequences were not assessed. Validity and feasibility evidence in this study indicate that the use of the objective structured clinical examination scores can be a valid way to assess CanMEDS competencies required for independent practice in pediatric critical care medicine.

Collecting Validity Evidence for the Assessment of Mastery Learning in Simulation-Based Ultrasound Training.

To collect validity evidence for the assessment of mastery learning on a virtual reality transabdominal ultrasound simulator. We assessed the validity evidence using Messick's framework for validity. The study included 20 novices and 9 ultrasound experts who all completed 10 obstetric training modules on a transabdominal ultrasound simulator that provided automated measures of performance for each completed module (i. e., simulator metrics). Differences in the performance of the two groups were used to identify simulator metrics with validity evidence for the assessment of mastery learning. The novices continued to practice until they had attained mastery learning level. One-third of the simulator metrics discriminated between the two groups. The median simulator scores from a maximum of 40 metrics were 17.5 percent (range 0 - 45.0 percent) for novices and 90.0 percent (range 85.0 - 97.5) for experts, p < 0.001. Internal consistency was high, with a Cronbach's alpha value of 0.98. The test/retest reliability gave an intra-class correlation coefficient (ICC) of 0.62 for novices who reached the mastery learning level twice. Novices reached the mastery learning level within a median of 4 attempts (range 3 - 8) corresponding to a median of 252 minutes of simulator training (range 211 - 394 minutes). This study found that validity evidence for the assessment of mastery learning insimulation-based ultrasound training can be demonstrated and that ultrasound novices can attain mastery learning levels with less than 5 hours of training. Only one-third of the standard simulator metrics discriminated between different levels of competence.

A Case for Caution: Chart-Stimulated Recall

This article reviews theoretical advantages of chart- stimulated recall (CSR), explores threats to validity due to construct underrepresentation and construct irrelevant variance using Messick's framework, and discusses possible solutions. The results can inform the GME community on considerations and potential solutions when implementing CSRs as part of an assessment system. We also identify areas for future research studies. Chart-Stimulated Recall CSR is a hybrid assessment format that combines chart review and an oral examination, with both based on a clinician's documented patient encounter. Faculty or the learner selects the clinical chart for a learner's patient to be used as a stimulus for questioning. 8-16 Using the learner's own clinical chart situates the examination within a realistic context, adding to the authenticity and value of the exercise. 17 Through a series of probing questions designed to inquire into the learner's clinical decision-making skills, the examinee is asked to reflect on and explain his or her rationale for clinical decisions. CSR has been used extensively in the United Kingdom and in Canada for the assessment of practicing physicians; in the United States it is predominantly used to assess trainees. A variety of scoring forms have been developed for CSR, ranging from checklists with comment boxes to ordinal rating scales. 11,18,19 Feedback usually is given to the learner at the end of the encounter 11,20 and may include action plans to improve future clinical decision making. 11,18,20-22 Despite evidence to sup- port the use of CSR in assessing the competence of practicing physicians, its use for certification of physicians has diminished due to practical concerns, such as cost, time, and the need for experienced assessors. 10,11,16,23

The IDEA Assessment Tool: Assessing the Reporting, Diagnostic Reasoning, and Decision-Making Skills Demonstrated in Medical Students' Hospital Admission Notes

Construct: Clinical skills are used in the care of patients, including reporting, diagnostic reasoning, and decision-making skills. Written comprehensive new patient admission notes (H&Ps) are a ubiquitous part of student education but are underutilized in the assessment of clinical skills. The interpretive summary, differential diagnosis, explanation of reasoning, and alternatives (IDEA) assessment tool was developed to assess students' clinical skills using written comprehensive new patient admission notes. Background: The validity evidence for assessment of clinical skills using clinical documentation following authentic patient encounters has not been well documented. Diagnostic justification tools and postencounter notes are described in the literature1,2 but are based on standardized patient encounters. To our knowledge, the IDEA assessment tool is the first published tool that uses medical students' H&Ps to rate students' clinical skills. Approach: The IDEA assessment tool is a 15-item instrument that asks evaluators to rate students' reporting, diagnostic reasoning, and decision-making skills based on medical students' new patient admission notes. This study presents validity evidence in support of the IDEA assessment tool using Messick's unified framework, including content (theoretical framework), response process (interrater reliability), internal structure (factor analysis and internal-consistency reliability), and relationship to other variables. Results: Validity evidence is based on results from four studies conducted between 2010 and 2013. First, the factor analysis (2010, n = 216) yielded a three-factor solution, measuring patient story, IDEA, and completeness, with reliabilities of .79, .88, and .79, respectively. Second, an initial interrater reliability study (2010) involving two raters demonstrated fair to moderate consensus (κ = .21–.56, ρ =.42–.79). Third, a second interrater reliability study (2011) with 22 trained raters also demonstrated fair to moderate agreement (intraclass correlations [ICCs] = .29–.67). There was moderate reliability for all three skill domains, including reporting skills (ICC = .53), diagnostic reasoning skills (ICC = .64), and decision-making skills (ICC = .63). Fourth, there was a significant correlation between IDEA rating scores (2010–2013) and final Internal Medicine clerkship grades (r = .24), 95% confidence interval (CI) [.15, .33]. Conclusions: The IDEA assessment tool is a novel tool with validity evidence to support its use in the assessment of students' reporting, diagnostic reasoning, and decision-making skills. The moderate reliability achieved supports formative or lower stakes summative uses rather than high-stakes summative judgments.

Technical Skills Assessment Toolbox

The purpose of this study was to create a technical skills assessment toolbox for 35 basic and advanced skills/procedures that comprise the American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) surgical skills curriculum and to provide a critical appraisal of the included tools, using contemporary framework of validity. Competency-based training has become the predominant model in surgical education and assessment of performance is an essential component. Assessment methods must produce valid results to accurately determine the level of competency. A search was performed, using PubMed and Google Scholar, to identify tools that have been developed for assessment of the targeted technical skills. A total of 23 assessment tools for the 35 ACS/APDS skills modules were identified. Some tools, such as Operative Performance Rating System (OSATS) and Objective Structured Assessment of Technical Skill (OPRS), have been tested for more than 1 procedure. Therefore, 30 modules had at least 1 assessment tool, with some common surgical procedures being addressed by several tools. Five modules had none. Only 3 studies used Messick's framework to design their validity studies. The remaining studies used an outdated framework on the basis of "types of validity." When analyzed using the contemporary framework, few of these studies demonstrated validity for content, internal structure, and relationship to other variables. This study provides an assessment toolbox for common surgical skills/procedures. Our review shows that few authors have used the contemporary unitary concept of validity for development of their assessment tools. As we progress toward competency-based training, future studies should provide evidence for various sources of validity using the contemporary framework.

External validation of scoring instruments for evaluating pediatric resuscitation.

Although many methods have been proposed to assess clinical performance during resuscitation, robust and generalizable metrics are still lacking. Further research is necessary to develop validated clinical performance assessment tools and show an improvement in outcomes after training. We aimed to establish evidence for validity of a previously published scoring instrument--the Clinical Performance Tool (CPT)--designed to evaluate clinical performance during simulated pediatric resuscitations. This was a prospective experimental trial performed in the simulation laboratory of a pediatric tertiary care facility, with a pretest/posttest design that assessed residents before and after pediatric advanced life support (PALS) certification. Thirteen postgraduate year 1 (PGY1) and 11 PGY3 pediatric residents completed 5 simulated pediatric resuscitation scenarios each during 2 consecutive sessions; between the 2 sessions, they completed a full PALS certification course. All sessions were video recorded. Sessions were scored by raters using the CPT; total scores were expressed as a percentage of maximum points possible for each scenario. Validity evidence was established and interpreted according to Messick's framework. Evidence regarding relations to other variables was assessed by calculating differences in scores between pre-PALS and post-PALS certification and PGY1 and PGY3 using a repeated-measures analysis of variance test. Internal structure evidence was established by assessing interrater reliability using intraclass correlation coefficients (ICCs) for each scenario, a G-study, and a variance component analysis of individual measurement facets (scenarios, raters, and occasions) and associated interactions. Overall scores for the entire study cohort improved by 10% after PALS training. Scores improved by 9.9% (95% confidence interval [CI], 4.5-15.4) for the pulseless nonshockable arrest (ICC, 0.85; 95% CI, 0.74-0.92), 14.6% (95% CI, 6.7-22.4) for the pulseless shockable arrest (ICC, 0.98; 95% CI, 0.96-0.99), 4.1% (95% CI, -4.5 to 12.8) for the dysrhythmias (ICC, 0.92; 95% CI, 0.87-0.96), 18.4% (95% CI, 9.7-27.1) for the respiratory scenario (ICC, 0.97; 95% CI, 0.95-0.98), and 5.3% (95% CI, -1.4 to 2.0) for the shock scenarios (ICC, 0.94; 95% CI, 0.90-0.97). There were no differences between PGY1 and PGY3 scores before or after the PALS course. Reliability of the instrument was acceptable as demonstrated by a mean ICC of 0.95 (95% CI, 0.94-0.96). The G-study coefficient was 0.94. Most variance could be attributed to the subject (57%). Interactions between subject and scenario and subject and occasion were 9.9% and 1.4%, respectively, and variance attributable to rater was minimal (0%). Pediatric residents improved scores on CPT after completion of a PALS course. Clinical Performance Tool scores are sensitive to the increase in skills and knowledge resulting from such a course but not to learners' levels. Validity evidence from scores for the CPT confirms implementation in new contexts and partially supports internal structure. More evidence is required to further support internal structure and especially to support relations with other variables and consequence evidence. Additional modifications should be made to the CPT before considering its use for high-stakes certification such as PALS.

고등학생들의 융합에 대한 태도 검사도구의 개발과 타당화

이 연구에서는 고등학생들의 융합에 대한 태도를 측정하는 검사도구를 개발하고 타당도를 확인하였다. 문헌연구를 통해 고등학생들의 융합에 대한 태도의 5가지 구인 (융합에 대한 지식, 개인적 관련성, 사회적 관련성, 흥미, 자아효능감)을 설정하였고, 각 구인들을 측정할 수 있는 예비문항 개발하여 233명의 고등학교 2학년 자연계열 학생들에게 투입하였다. 문항 타당도는 Messick (1995)이 제시한 내용에 기초한 타당도, 실제에 기초한 타당도, 구조에 기초한 타당도에 기초하였으며, 전문가 검토, 라쉬 분석, 구조방정식을 활용하여 확인하였다. 연구 결과 최종적으로 선정된 5개 구인 23개 문항은 타당한 문항으로 확인되었다. 또한 융합에 대한 태도는 과학 동기를 향상 시킬 수 있다는 이론적 모델 역시 구조방정식으로 확인하였고 타당한 모델로 확인되었다. 이 연구를 통하여 개발된 문항은 학생들의 융합에 대한 태도와 융합 과학 학습의 효과를 확인하는데 활용 될 수 있을 것이다. This study aims to develop and validate an instrument to measure students' attitude toward convergence. To do so, we have defined five constructs (i.e. knowledge about convergence, personal relevance, social relevance, interest and self-efficacy) of 'attitude toward convergence' based on literature review, developed items, and collected data from 233 11th grade science track students. The validity of these items have been evaluated by Messick's framework (1995) (i.e. content, substantive, structural aspects of validity), experts' review, Rasch analysis, and confirmatory factor analysis using structural equation modeling. Our results have confirmed the five constructs and 23 selected items have met the benchmark of item validity. Moreover, the theoretical model illustrating that the high level of attitude toward convergence increases the level of science motivation has also been supported by the data. The items developed in this study will be used to measure students' attitude toward convergence and to estimate the effect of learning program for convergence science.

Technology-Enhanced Simulation to Assess Health Professionals

To summarize the tool characteristics, sources of validity evidence, methodological quality, and reporting quality for studies of technology-enhanced simulation-based assessments for health professions learners. The authors conducted a systematic review, searching MEDLINE, EMBASE, CINAHL, ERIC, PsychINFO, Scopus, key journals, and previous reviews through May 2011. They selected original research in any language evaluating simulation-based assessment of practicing and student physicians, nurses, and other health professionals. Reviewers working in duplicate evaluated validity evidence using Messick's five-source framework; methodological quality using the Medical Education Research Study Quality Instrument and the revised Quality Assessment of Diagnostic Accuracy Studies; and reporting quality using the Standards for Reporting Diagnostic Accuracy and Guidelines for Reporting Reliability and Agreement Studies. Of 417 studies, 350 (84%) involved physicians at some stage in training. Most focused on procedural skills, including minimally invasive surgery (N=142), open surgery (81), and endoscopy (67). Common elements of validity evidence included relations with trainee experience (N=306), content (142), relations with other measures (128), and interrater reliability (124). Of the 217 studies reporting more than one element of evidence, most were judged as having high or unclear risk of bias due to selective sampling (N=192) or test procedures (132). Only 64% proposed a plan for interpreting the evidence to be presented (validity argument). Validity evidence for simulation-based assessments is sparse and is concentrated within specific specialties, tools, and sources of validity evidence. The methodological and reporting quality of assessment studies leaves much room for improvement.

A Psychometric Validation of the Beliefs Toward Mental Illness Scale

This study measured Protestant Christians' beliefs about mental illness using the Beliefs Toward Mental Illness Scale (BMI). This study used a state-of-the-art item response theory technique, namely the Rasch rating scale model (RRSM), to evaluate the psychometric properties of the BMI and to produce a ruler of the construct hierarchy. Using Messick's framework for evaluating construct validity, sufficient evidence was obtained that suggests the BMI is psychometrically sound regarding substantive, content, structural, and generalizable aspects of validity. The BMI is a psychometrically sound instrument that is appropriate for measuring beliefs toward mental illness.

Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks

Because learning and instruction are increasingly competence-based, the call for assessment methods to adequately determine competence is growing. Using just one single assessment method is not sufficient to determine competence acquisition. This article argues for Competence Assessment Programmes (CAPs), consisting of a combination of different assessment methods, including both traditional and new forms of assessment. To develop and evaluate CAPs, criteria to determine their quality are needed. Just as CAPs are combinations of traditional and new forms of assessment, criteria used to evaluate CAP quality should be derived from both psychometrics and edumetrics. A framework of 10 quality criteria for CAPs is presented, which is then compared to Messick's framework of construct validity. Results show that the 10-criterion framework partly overlaps with Messick's, but adds some important new criteria, which get a more prominent place in quality control issues in competence-based education.

A Construct Validation Study of Phonological Awareness for Children Entering Prekindergarten

The purpose of this study was to determine the psychometric characteristics of a phonological awareness assessment for prekindergarten children using Messick's (1989) framework for unitary construct validity. Upon entry into prekindergarten, children were given rhyme discrimination, syllable segmentation, initial phoneme isolation, and phoneme blending assessments drawn from The Phonological Awareness Test (Robertson &amp; Salter, 1997). Item analyses indicated high internal consistencies, but some tasks were fairly difficult for the age group. Confirmatory factor analysis revealed that the four phonological awareness tasks have two correlated factors that underlie the measurement (large unit and small unit tasks). Multiple regression analysis showed that all four tasks had significant predictive validity for alphabetic knowledge. Given the paucity of standardized phonological awareness tests for children under 5 and the increased attention to preliteracy assessment, this off-level use of the test has the advantage of providing an ongoing examination of children's early and later phonological awareness development using the same instrument.

Validity and Reliability of Portfolio Assessment of Competency in a Baccalaureate Dental Hygiene Program

This study examined the validity and reliability of portfolio assessment using Messick's unified framework of construct validity. Theoretical and empirical evidence was sought for six aspects of construct validity. Seven faculty raters evaluated twenty student portfolios using a primary trait analysis scoring rubric. A significant relationship (r = .81-.95; p < .01) between the seven subscales in the scoring rubric demonstrates measurement of a common construct. There was a significant relationship between portfolios and GPA (r = .70; p < .01) and the NBDHE (r = .60; p < .01). The relationship between portfolios and the Central Regional Dental Testing Service (CRDTS) examination was both weak and nonsignificant (r = .19; p > .05). A fully crossed, two-facet generalizability (G) study design was used to examine reliability. ANOVA demonstrated that the greatest source of variance was the scoring rubric itself, accounting for 78 percent of the total variance. The smallest source of variance was the interaction between portfolio and rubric (1.15 percent). Faculty rater variance accounted for only 1.28 percent of total variance. A phi coefficient of .86, analogous to a reliability coefficient in classical test theory, was obtained in the decision study by increasing the subscales to fourteen and decreasing faculty raters to three. In conclusion, the pattern of findings from this study suggests that portfolios can serve as a valid and reliable measure for assessing student competency.

Issues in the Validation of Assessment in Technology-Based Distance and Distributed Learning: What Can We Learn From Messick's Framework?

In his (1988) article on validity and validation, Messick discussed how a proliferation of technology-mediated delivery methods in the first few decades of the 21st century will impact on our conceptions of teaching and learning, and on validity and validation practices. According to Messick (1988), our fundamental conception of validity, as expressed in his four-faceted framework, will likely remain the same, but new technology-mediated environments will render a classical, unitary approach to validation untenable. With the aid of an example, this article demonstrates how Messick's framework provides a powerful set of lenses with which to explore issues in the validation of small-scale assessments in these new environments. A goal of this article is to initiate a dialogue between measurement and testing specialists and experts in distance and distributed learning.

The unified validity of the Four Skills Exam: applying Messick's framework

The unified validity of the Four Skills Exam: applying Messick's framework

The unified validity of the Four Skills Exam: applying Messick's framework

In the USA, 17 states use Spanish-language proficiency tests to ensure that bilingual education teachers are able to deliver academic instruction in Spanish to school-age students. However, little is known about the tests’ validity. In this study the unified validity of the Four Skills Exam (FSE), used in New Mexico for nearly 18 years, was evaluated using Messick’s framework (1989). Through this analysis, the FSE was found to lack an adequate degree of validity due to questionable psychometric properties and factors external to the test. Messick’s framework offers bilingual educators a comprehensive blueprint for assessing the validity of these tests. The review of the FSE also elucidates pitfalls in bilingual education teacher language proficiency testing which other states should avoid.