A Substantive Process Analysis of Responses to Items from the Multistate Bar Examination

Abstract

We investigated examinees' cognitive processes while they solved selected items from the Multistate Bar Exam (MBE), a high-stakes professional certification examination. We focused on ascertaining those mental processes most frequently used by examinees, and the most common types of errors in their thinking. We compared the relationships between different problem-solving processes, errors, and performance, and assessed the degree to which performance was related to correspondence between examinees' and model problem-solving approaches. Participants most commonly solved problems with reference to legal principles and cues presented in problem stems, but also resorted to other mental processes. We found that performance on the MBE is strongly related to domain-specific skill in using legal principles, as well as to domain-general metacognitive skill in organizing thinking. Evidence that would suggest that testwiseness strategies or common-sense reasoning was a source of construct-irrelevant variance on the MBE was not found. Our study demonstrated a systematic method for in-depth analysis of examinee substantive processes on examinations such as the MBE that assess complex reasoning within a specific domain.