Method and model comparison by sum of ranking differences in cases of repeated observations (ties)

Abstract

Significant progress has been achieved since the introduction of the new similarity measure: the sum of absolute ranking differences (SRDs) [TrAC — Trends in Anal. Chem. 29 (2010) 101–109]. Empirical evidences were accumulated about scaling, selection of the reference (benchmark) vector, cross-validation and grouping of variables (features, models, methods, etc.). The theory has been developed including the repeated observations (ties):(i)The exact theoretical distribution (null distribution) for 4<number of objects<9 has been calculated for SRD treatment of random numbers (it's a kind of validation, a permutation test). All possible reference vectors with ties imply different distribution.(ii)For number of objects above eight (n>8) an approximation has been developed using the Gaussian distribution fitted on the SRD distribution given by generating of three million n-dimensional random vectors.The validity and features of the SRD methodology with ties are illustrated using two case studies: evaluation of a sensory panel and ranking of financial indicators.