Rank analysis in evaluating the validity of olympiad tasks

Abstract

The rank analysis (cenological approach) based on a strict mathematical apparatus allows diagnosing the state of educational systems and processes on a scientific objective basis. The rank analysis base is applying the law of hyperbolic rank distribution of objects in systems-cenoses, which is a refined Pareto law 20/80 (Pareto-Kudrin law). Using of rank analysis to evaluate the validity of the olympiad tasks of the all-Russian Multidisciplinary Engineering Olympiad «Star», the authors demonstrate the necessity of implementing rank analysis in the practice of education quality management. They show that this law is important and appropriate to apply in the education quality management. Optimization of educational systems and processes using rank analysis consists in eliminating anomalous deviations from the hyperbolic law, which is possible only with graphical visualization of the rank distribution and its approximation. The rank distribution graph provides visibility and insight into the nature of the rank decrease. Shortcomings in the content of olympiad tasks (as well as tests), leading to a deterioration of their validity and reliability, are shown the form of distortions of the hyperbolic graph (humps, depressions, tails, degeneration of the hyperbola into other dependencies). The rank analysis allows identifying objectively several levels of validity of olympiad or test tasks carrying on a scientific basis. The paper demonstrates graphs of run distributions illustrating examples of high (natural science) and below the average (history) validity of olympiad tasks. Educational systems are cenoses, ranking objects in them are students, classes, schools, etc., and their ranked parameters are academic performance, rating in points, performance indicators, etc. Using simple mathematical tools of rank analysis and computer software bring the technologies to assess the education quality to a higher level meeting the challenges of the time.