Investigating The Effect Of Testlets Consisting Of Open-Ended And Multiple-Choice Items On Reliability Via Generalizability Theory

Abstract

This study aimed to reveal the effect of testlets consisting of open-ended and multiple-choice items with similar content on reliability. For this purpose, the Mathematics Achievement Test with four testlets, one of which consisted of open-ended items and the other of multiple-choice items, was administered to 128 8th-grade students. Reliability estimations on the obtained data were conducted in the Edu-G program based on the Generalizability Theory. A decision study was also performed in the study. In the achievement test with testlets consisting of open-ended items, p×i×r (p: person, i: item, r: rater) fully crossed design was used when testlet effect was not considered; p×(i:t)×r (t: testlet) nested design was used when testlet effect was considered. According to the results obtained, the reliability coefficient was estimated higher when the testlet effect was not considered. Similarly, in the achievement test with testlets consisting of multiple-choice items, the p×i crossed design was used when the testlet effect was not considered, and the p×(i:t) nested design was used when the testlet effect was considered. According to the results, the reliability coefficient was similarly estimated higher when the testlet effect was not considered. According to the data obtained within the scope of the study, the reliability coefficient was estimated according to the treatment of the testlet effect in the test with open-ended items.