Automatic performance assessment in Virtual Reality medical simulators: A model based on procedure trajectories and machine learning

Abstract

Virtual Reality (VR) simulators for medical training helps students develop the sensorimotor skills required to perform specific procedures. However, as a training resource, a VR simulator usually lacks the capacity to assess the user’s performance automatically and provide feedback. The main objective of this study is to develop a model to automatically assess performance in VR medical simulators, based on the medical instrument trajectories and machine learning. We designed a model to use the data collected in medical VR simulators and defined two different labeling systems: by level of expertise or by a dental instructor assessment. We calculated 98 features related to the participant’s performance and combined three different feature selection/fusion algorithms with five classifiers. The SVM algorithm accomplished the best results overall, with an accuracy of 0.77, specificity of 0.60 and sensitivity of 0.94 using the labeling by the dental instructor’s assessment. Overall, the results for the performance assessment were promising and the model can be trained for similar VR medical simulators that collects trajectory data through a haptic device.