Abstract
BackgroundThe validity of high-stakes decisions derived from assessment results is of primary concern to candidates and certifying institutions in the health professions. In the field of orthopaedic manual physical therapy (OMPT), there is a dearth of documented validity evidence to support the certification process particularly for short-answer tests. To address this need, we examined the internal structure of the Case History Assessment Tool (CHAT); this is a new assessment rubric developed to appraise written responses to a short-answer test of clinical reasoning in post-graduate OMPT certification in Canada.MethodsFourteen physical therapy students (novices) and 16 physical therapists (PT) with minimal and substantial OMPT training respectively completed a mock examination. Four pairs of examiners (n = 8) participated in appraising written responses using the CHAT. We conducted separate generalizability studies (G studies) for all participants and also by level of OMPT training. Internal consistency was calculated for test questions with more than 2 assessment items. Decision studies were also conducted to determine optimal application of the CHAT for OMPT certification.ResultsThe overall reliability of CHAT scores was found to be moderate; however, reliability estimates for the novice group suggest that the scale was incapable of accommodating for scores of novices. Internal consistency estimates indicate item redundancies for several test questions which will require further investigation.ConclusionFuture validity studies should consider discriminating the clinical reasoning competence of OMPT trainees strictly at the post-graduate level. Although rater variance was low, the large variance attributed to error sources not incorporated in our G studies warrant further investigations into other threats to validity. Future examination of examiner stringency is also warranted.
Highlights
The validity of high-stakes decisions derived from assessment results is of primary concern to candidates and certifying institutions in the health professions
The approaches to orthopaedic manual physical therapy (OMPT) training are similar between pre-licensure and post-graduate education contexts in Canada, clinical reasoning processes observed in novice physical therapists have been characterized primarily by hypothetico-deductive reasoning and differ from the diverse reasoning processes employed by physical therapists with greater expertise [20,21,22]
The assessment of clinical reasoning is context specific and demand adequate sampling in order to provide an appropriate assessment of this construct [29]; while results from this study offer evidence of validity to support the use of scores derived from the Case History Assessment Tool (CHAT), other assessment formats are needed to triangulate data concerning candidates’ clinical reasoning competence
Summary
The validity of high-stakes decisions derived from assessment results is of primary concern to candidates and certifying institutions in the health professions. In the field of orthopaedic manual physical therapy (OMPT), there is a dearth of documented validity evidence to support the certification process for short-answer tests. To address this need, we examined the internal structure of the Case History Assessment Tool (CHAT); this is a new assessment rubric developed to appraise written responses to a short-answer test of clinical reasoning in post-graduate OMPT certification in Canada. Evidence of validity can be generated from five distinct sources: instrument content; response process; internal structure of the instrument; relationships between assessment scores and other variables; and the consequences of decisions made based on the assessment results (Table 1) [5]
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
