Constructing and evaluating a validity argument for the final-year ward simulation exercise.

Hettie Till,Kevin Stirling,Jean Ker,Gary Mires,Carol Myford

doi:10.1007/s10459-015-9601-5

Abstract

The authors report final-year ward simulation data from the University of Dundee Medical School. Faculty who designed this assessment intend for the final score to represent an individual senior medical student's level of clinical performance. The results are included in each student's portfolio as one source of evidence of the student's capability as a practitioner, professional, and scholar. Our purpose in conducting this study was to illustrate how assessment designers who are creating assessments to evaluate clinical performance might develop propositions and then collect and examine various sources of evidence to construct and evaluate a validity argument. The data were from all 154 medical students who were in their final year of study at the University of Dundee Medical School in the 2010-2011 academic year. To the best of our knowledge, this is the first report on an analysis of senior medical students' clinical performance while they were taking responsibility for the management of a simulated ward. Using multi-facet Rasch measurement and a generalizability theory approach, we examined various sources of validity evidence that the medical school faculty have gathered for a set of six propositions needed to support their use of scores as measures of students' clinical ability. Based on our analysis of the evidence, we would conclude that, by and large, the propositions appear to be sound, and the evidence seems to support their proposed score interpretation. Given the body of evidence collected thus far, their intended interpretation seems defensible.

Highlights

The valid and reliable assessment of clinical performance at any stage of medical practice is important for the reassurance of patients and for the accountability of the profession (General Medical Council, 2010)
We use as our example the final-year Ward Simulation Exercise (FYWSE), created by faculty and the University of Dundee Medical School to assist them in determining whether their senior medical students have acquired the level of clinical ability needed to provide high quality patient care
The data we used in our analyses demonstrated acceptable fit to the multifacet Rasch measurement (MFRM) model, and our results showed that the FYWSE could reliably (.96) separate the students into distinct levels of clinical ability, giving us reassurance that the FYWSE was able to identify those students who are competent in the clinical setting, as well as those who still needed some practice

Summary

Introduction

The valid and reliable assessment of clinical performance at any stage of medical practice is important for the reassurance of patients and for the accountability of the profession (General Medical Council, 2010). Providing sound validity evidence to support their claim that students have demonstrated this level of clinical ability can be challenging (Epstein, 2007). The assessment designers must develop validity arguments that support (or refute) each intended interpretation of scores for a specified use. That is, they must determine which type(s) of validity evidence (e.g., evidence based on test content, response processes, internal structure, relations to other variables, and/or consequences of testing) are most needed for evaluating the soundness of each proposition, and collect and examine that evidence to determine to what extent it supports the proposed score interpretation

Methods

Results

Discussion

Conclusion