Initial validation of a script concordance test to measure the development of clinical reasoning among physical therapy residents

Abstract

Rationale: Clinical reasoning (CR) skills of physical therapy (PT) residents may be limited by a lack of clinical experience, and the complexity of neurologic and elderly patients requires individualized approaches for prescribing walking assistive devices (ADs). Script concordance tests (SCTs), which are case-based assessments that present information that the test taker must examine before answering, can measure the growth in CR. The purpose was to validate a SCT designed to measure CR growth in residents for walking AD prescription. Methods: Using a prospective cohort design, a panel of 20 experienced practicing clinicians reviewed the test content and provided the correct answers. Neurologic and geriatric residencies were identified from the American Board of Physical Therapy Residencies and Fellowships database. Residency directors were informed of the study, and 27 of them provided access to their residents. Residents gave assent in month 1 and were followed through month 6. The SCT and instructions were emailed at both time points. In total, 27 month-1 tests and 20 month-6 tests were returned. Fleiss kappa was used to evaluate the consistency of the clinician panel’s and the residents’ responses at both time periods. Mean scores for the panel and the residents at months 1 and 6 were calculated. Testing for differences between the clinician panel and the residents at months 1 and 6 was assessed using a Mann-Whitney U test, and testing for differences between the residents at the same time points was assessed using a Wilcoxon signed rank test. Results: Demographics of the clinical panel and residents are presented. Internal consistency was moderate, and chance agreement was fair in both groups. Mean test scores were 34.8 (4.3), 34.5 (4.1), and 36.3 (3.9) points for the panel, and residents at months 1 and 6, respectively. Concordance of choices on the SCT between the clinician panel and residents was similar at month 1 and grew more concordant by month 6. Conclusion: The SCT did not differentiate between the practicing clinicians’ and residents’ CR skills to prescribe ADs. Further research comparing the CR of PT residents to PT clinicians in early practice should be undertaken since the residents’ responses were consistent with the practicing clinicians. The unique test construction or the clinician panel, whose members were not required to be board certified, may have introduced error.