An analysis of practice effects caused by different item orders in English grammar tests

Abstract

This study aims to examine the practice effects that can occur when multiple items assessing the same knowledge are presented on a in-school test that is usually based on a specific textbook with designated units and thus can have multiple items asking for the same knowledge/skill. For this purpose, the study analyzes the response data from two college English grammar tests administered in two semesters. The grammar tests were composed of two different sets (A and B) that had different item orders. The focus of the analysis was on the pairs of items that measure the same skill and appear in the opposite order in the two test sets, A and B. Applying the chi-square test to find out if there was any statistically significant different proportion correct (item facility) on each of these items across different test sets. The study found that the response data from one semester had more of these items with different item facility across the test sets, which indirectly implies the practice effect of these items. The paper concludes with suggestions for future research based on the limitations of the study.