Modelling experts' behavior with e-valUAM to measure computer science skills

Abstract

In this work we present an evaluation method that focuses on experts' behavior instead of the traditional scores based just on the number of correct answers. The method presented here is especially suitable to measure the skills in Computer Science since this is a wide discipline very difficult to evaluate due to the many facts publicly available on the Internet. By using traditional evaluation tools, it is very difficult to measure the real knowledge of the users since they can correctly answer even without having acquired formal academic knowledge. To use this method, we have developed a test that can detect significant differences between standard users and experts in Computer Science. The test is applied by the e-valUAM application, which has been modified to store several parameters from the users' answers. By optimizing the parameters by a linear model, we have developed an equation that can be used to quantitatively compare the results of a single user with the results from the reference group of experts. This optimization is only possible because this group shows good stability and gives statistically different results compared to the other groups. The scores achieved with our method can be used to predict the formal knowledge of the users and modify their training when needed.