Establishing Physics Concept Inventories Using Free-Response Questions

Abstract

Concepts are important building blocks for understanding complicated topics and entire disciplines such as physics. The idea of an inventory of concepts has been proposed as the basis for investigating the readiness of students to develop their understanding. Hestenes et al. introduced concept inventories in physics using the multiple-choice question format. There is interest in using a less-constrained free-response format with computer-automated marking to enable more efficient use of concept inventories. The adaptation of Hestenes’ Force Concept Inventory (FCI) for use with free-response, computer-marked format is the subject of this thesis. This study establishes the Alternative Mechanics Survey (AMS), a free-response computer-marked version of the FCI and validates it for use as an alternative to the multiple-choice FCI. The AMS was subject to validity testing with a pilot group through usability tests followed by semi-structured interviews, which were analyzed using Thematic Analysis. This established that the AMS structure involving the free-response format was viable. Classical Test Theory (CTT) was used to test for reliability of the AMS questions. Data from 335 completed attempts were analyzed to generate Kuder-Richardson reliability and Ferguson’s delta statistics which showed the questions to be acceptable. The AMS marking rules were also tested for reliability by calculating Inter-Rater Reliability (IRR) statistics for 8091 question responses. The calculated values of the Cohen’s kappa statistic showed that the marking rules were acceptable. This work has demonstrated validity of the AMS with the free-response format, and the reliability of the specific question set posed together with the reliability of the associated marking rules. To demonstrate the applicability of the approach in another domain, the General Relativity Concept Inventory (GRCI), a new free-response concept inventory covering basic General Relativity concepts has been developed and tested. It is concluded that physics concept inventories can be established using free-response questions.