Model for evaluation of multiple abilities programming problems in online massive environments

Abstract

Research indicates human being is endowed with multiple intelligences, skills, and abilities. In the context of education, many exercises demand multiple skills from students to successfully solve them in different areas of knowledge. In computer science, computer programming is one of the skills that involves the use of multiple skills for problem-solving, where problems can be solved in more than one way (paths). On massive environments for teaching programming, it is common for automatic assessment systems to observe only the final result of the student's interaction with the learning object, not identifying the individual interaction of multiple skills needed to solve the problem nor identifying a solution path adopted by the student. Many models were proposed based on Elo models, which use performance expectation, and Item Response Theory, but these models do not consider the various paths to solve problems. The objective of this work is to propose a model also based on performance expectation, which individually estimates multiple abilities of students in the context of massive online education, assuming problems have more than one solution, and there is access only to the final result (right or wrong). An experimental setup is proposed to validate the model, involving the use and analysis of the proposed model through an experiment in a database, named beecrowd, and a case study with programming students. Model results are satisfactory, since: i) it is possible to treat the student's abilities individually, as well as to follow the evolution of each ability over time; ii) it is possible to predict the paths adopted by them according to the student's abilities; iii) the model also shows positive results when integrated with a recommendation system, recommending problems compatible with the student's abilities.