Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games

Abstract

In learning environments, appropriate objectives are needed to create the conditions for learning and consequently the performance to occur. It follows that appropriate metrics would also be necessary to properly measure what actually constitute performance in situ (within that environment), and to measure if learning has indeed occurred. Serious games environments can be problematic for performance measurement because publishers often posit the game would automatically facilitate learning by their design. Stakeholders, on the other hand, require empirical proofs to quantify performance improvement and calculate Returns of Investment.Serious games environment (an open-ended scenario) with ‘more-than-one correct solutions’ can be difficult for data analysis. In a previous study, we demonstrated the possible use of String Similarity Index to differentiate novices from experts based on how (dis-)similar their performances are within a ‘single-solution’ serious game environment. This study extends the previous study by differentiating a group of novices from the experts based on how (dis)similar their performances are within a ‘multiple-solution’ serious game environment. To facilitate the calculation of performance, we create a new metric for this purpose called, Maximum Similarity Index, to take into consideration the existence of multiple expert solutions. Our findings indicated that Maximum Similarity Index can be a useful metric for serious games analytics when such scenarios present themselves, both for the differentiation of novices from experts, and for the ranking of the player cohort. In a secondary analysis, we compared Maximum Similarity Index to other commonly available game metrics (such as time of completion) and found it to be more appropriate than other game metrics for the measurement of performance in serious games.