Use of artificial intelligence methods to assess competencies during testing

Abstract

According to modern standards of higher education (FSES 3++), teaching is aimed at developing a set of universal, general professional, and professional competencies in a graduate. In this regard, the problem of assessing the level of their formation arises. Since competence is a multidisciplinary concept, its assessment requires designing a task that simulates a real situation, which may require complex knowledge and skills related to different areas of knowledge. One of the modern tools for current and final control is computer testing. In this case, during one testing session, it is desirable to evaluate several competencies at once with a limited number of tasks. The task is relevant, because in the currently used classical one-dimensional one-, two- and three-parameter Rasch—Birnbaum models, it is assumed that all tasks in the test belong to the same area of knowledge.The purpose of the article is to propose a methodology for the intellectual assessment of several competencies based on the results of one testing session, based on the modification of the Bayes algorithm, in which the choice of the next task is carried out as a result of analyzing the entropy (uncertainty) of the subject’s attribution to typical combinations (patterns) of preparedness levels for different competencies.An original technique for regularizing the Bayes algorithm based on the assessment of the change in the entropy of the probability distribution over patterns and the Kullback—Leibler divergence has been developed, which makes it possible to identify the moment when the student gives an answer that does not correspond to what is expected from them, based on the analysis of previous answers. The proposed technique not only increases the stability of the Bayes algorithm for measuring the probability of a student belonging to a given pattern by slightly increasing the required number of tasks but also allows to model the actions of an examiner in the process of competencies assessment. In general, the developed approach makes it possible to objectively assess the levels of formation of several competencies based on the results of single test.