Assessing Medical Students' Nontechnical Skills Using Immersive Simulation: What Are the Essential Components?

Abstract

Nontechnical skills (NTS) have been acknowledged to be important for medical students and can be linked to improved clinical performance. However, existing tools to evaluate these within a simulated setting address only a limited number of NTS. The Medical Students' Nontechnical Skills (Medi-StuNTS) behavioral marker system (BMS) outlines 5 categories of NTS for medical students. This study aimed to seek evidence for completeness and content validity to refine the BMS and to ascertain which NTS are essential for medical students. We asked 128 workshop participants if they felt there were any missing or irrelevant items in Medi-StuNTS system. A subject matter expert panel (n = 10) rated how essential they considered each item in the BMS. An Item-Content Validity Index was calculated for each skill element and the Scale-Content Validity Index was calculated as a measure of content validity of the full system. Of the workshop participants, 78.9% felt that there were no missing items and 93% felt that there were no irrelevant items. Potentially missing items highlighted were as follows: "working in a hierarchy," "leadership," "awareness of the emotional state of other team members," and "nonverbal communication." Fourteen of 16 skill elements achieved the recommended level for content validity (Item-Content Validity Index ≥ 0.78), and the Scale-Content Validity Index was higher than the acceptable level (≥0.8). Evidence for completeness and content validity of Medi-StuNTS has been demonstrated. There is a far wider range of NTS that seem to be essential for medical students than those assessed by tools developed before Medi-StuNTS. Medi-StuNTS provides comprehensive cover of the essential NTS required by medical students, with specific reference to the skill categories "self-awareness" and "escalating care," which do not feature in other tools for assessing NTS in this group.