Instructive head-mounted display system: pointing device using a vision-based finger tracking technique applied to surgical education.

Abstract

IntroductionThe head-mounted display (HMD) system is a novel personalized imaging monitoring system for use in a medical setting.AimTo support interactive intraoperative communication among HMD wearers, we integrated vision-based finger tracking into our system as a pointing device.Material and methodsOur vision-based finger tracking system is composed of a commercially available real-time video camera, which is mounted on the modern HMD, and computer software that enables tracking of the tip of the operator's index finger and superimposing the marker on the endoscopic view.ResultsWe used this system in an experimental demonstration. The operator used the finger-tracking pointer to explain the intraoperative findings of transurethral resection for bladder cancer to medical students.ConclusionsThis finger tracking system-based pointing device can function as a supportive tool for the HMD system, enabling interactive instruction and communication between the operator and other attending physicians or medical students.