Development of a Diagnostic Tool to Verify the Adequacy of Multiple-Choice Fill-in-the-Blank Programming Learning Tasks and the Appropriateness of Their Difficulty Level

Abstract

In programming education at higher education institutions, closed-ended assignments are often presented to learners. Closed-ended tasks are used as a means of checking the learner’s understanding level. A typical closed-ended problem in programming is a multiple-choice fill-in-the-blank learning task. When this learning task is applied to a class, the teacher must usually design the problem-solving procedure by themself in advance and verify its validity. Preventing unintended correct answers due to the teacher’s oversight is essential, but this task is not easy. Therefore, we developed a diagnostic tool in this study to verify the validityof multiple-choice fill-in-the-blanks programming learning tasks. Using the proposed tool, the teacher can enumerate patterns of input/output according to to answer combinations when creating questions. Since variations in inputs and outputs and overlaps of inputs and outputs are revealed in advance, the teacher can find the difficulty of the learning task and the existence of unintended correct answers before the question is posed. In this paper, we asked teachers to use the proposed tool on a trial basis and obtained feedback from them. As a result, the proposed tool revealed correct answers that the teachers had not noticed before. This defect was derived from a behavior peculiar to the C language and would have been difficult to detect under normal circumstances. Furthermore, the teacher could easily adjust the difficulty level of the problem by knowing the input/output variations results in advance. These results demonstrate the usefulness of the proposed tool.