Using Generalizability Analysis to Establish Guidelines for Designing Horizontally Integrated Anatomy Assessments

Abstract

Background & PurposeIn the context of planning assessments for a combined gross anatomy/histology course, this study used generalizability analysis to estimate the number of gross anatomy and histology items that would be needed on a series of examinations to reliably assess discipline‐specific knowledge.MethodsScores of first‐year medical students (n=145) who sat for 3 gross anatomy and histology written and practical examinations were analyzed. A generalizability study (G‐study) estimated the reliability of the written and practical examination scores, and a decision study (D‐study) estimated the optimum assessment parameters for reaching a minimum reliability of 0.8.ResultsEstimated G coefficients for calculating reliability were 蠅0.782 for the gross anatomy examinations and 蠅0.659 for the histology examinations. Assuming that a series of 4 separate examinations would be ideal for assessing a breadth of content in a combined gross anatomy/histology course, the D‐study estimated that a minimum of 35 and 40 items for a gross anatomy written and practical examination, respectively, would be needed to reach a reliability of 0.8 for measuring gross anatomy knowledge alone. Furthermore, each of the 4 examinations would require an additional 40 written and 45 practical items to reliability assess one's knowledge of histology.ConclusionThis study helps to establish basic parameters for designing horizontally integrated anatomy assessments. Future studies will investigate how incorporating ‘truly integrated items' (that require knowledge of histology and gross anatomy to answer a single question) may affect the reliability of such assessments.