QUESTIONS ABOUT QUESTIONS: EVALUATING THE QUALITY OF ASSESSMENTS IN THE DATA SCIENCE ERA

Abstract

INTRODUCTION: Assessments are not tools for merely grading students. They can be powerful instruments to monitor, measure, and improve both students learning and teaching process. In the Data Science era, educational technology has an important role collecting, storing and organizing data, allowing instructors to assess a large amount of students even remotely. Thus, once assessment is of central importance to education it is desired that these instruments are reliable and valid. OBJECTIVES: This work aims to describe the potential of validation tools united to educational technology and Rasch Analysis to improve assessments quality. MATERIALS AND METHODS: Three different virtual environments were used to collect data to this study: 3DClass, The Cell and ARMET, mobile applications able to track students’ performance answering quizzes. First, learning objectives were defined for each software purpose. Then, questions were created to assess students learning throughout quizzes. These questions were peerreviewed by experts and then applied to students. After that, the data were refined and analyzed using Rasch Model. DISCUSSION AND RESULTS: Results show that software used in this study provided detailed and organized data regarding students’ behavior, in large scale. From this data, the Rasch model identified overall reliability of the quizzes applied, pointing questions in need of reformulation, consequently identifying students’ abilities that were not estimated precisely. Moreover, we analyzed the assessments level of difficulty in relation to students sample in order to develop new questions that match them. From this data, instructors were able to reformulate the assessments to match course objectives and provide feedback to students in real time. CONCLUSION: This methodology allowed us to refine quizzes in every application to students, establishing criteria to choose which questions we would keep or remove from quizzes, improving their quality in terms of learning objectives and questions reliability.