Determining Ability Levels by Using Computerized Adaptive Testing and Paper-And-Pencil Testing in the Distance Education: A Multi-Centred Real Data Study

Abstract

Aim: With the popularity of distance education in the current Covid-19 pandemic, the use of computerized adaptive testing is becoming increasingly important in order to determine the ability level of individuals more accurately and to analyze the exam strategies in online exams. In this study, it is aimed to compare the estimations of students' ability levels in the distance education with computerized adaptive testing and paper-pencil test.Methods: This study is a cross-sectional and methodological study which was conducted between March 2020 and December 2020. The study population consists of 4th class students which were studyingphysical education and sports teaching at Erzincan Binali Yıldırım University, Gazi University, Erciyes University and Kastamonu University Sports Sciences Faculties. In this study, analyzes were performedby using R programming language (ver. 3.6.2) and RStudio (ver. 1.2.5033) software. Number of people was 176; the number of items in the question bank was 80 (formed with 20 questions each prepared byexperts from each of 4 different fields (exercise physiology, psychomotor development, training information, exercise and nutrition)); the standard error for the computerized adaptive test stoppingcriterion was taken as <0.5. The agreement of Paper-And-Pencil Test and computerized adaptive test θ values were evaluated with the intraclass correlation coefficient. CAT estimates of individuals wereperformed with the “randomCAT” function in the catR package. The ability level of a person was obtained with computerized adaptive test application, the standard error of θ and the number of items used were presented as a result of the analysis. Maximum Fisher information criterion was used for question selection in computerized adaptive test. Maximum likelihood estimation was used for iterative approaches in the ability level estimations.Results: There was moderate agreement between the classical test score (on a 0-100 scale) and the computer-adapted test (intraclass correlation coefficient: 0.682, confidence interval 95%CI=0.594-0.765, F(175.175)=17.1; p<0.001). In this study, the ability level estimations by using the item response theory from the answers obtained from the paper-and-pencil test and the ability level estimations with thecomputer adaptive test were examined. It was seen that the agreement was at an excellent level (intraclass correlation coefficient: 0.90; 95%CI: 0.866-0.924). Also it was also found that computerized adaptive test reduced the number of addressed questions by 80% when compared to Paper-And-Pencil Test. Conclusions: The results obtained from this study show that with the application of computer-adapted testing, safe results can be obtained in a shorter time, with less number of items, and the results can bemonitored shortly after the completion of the test.