The Deep Reinforcement Learning-Based VR Training System With Haptic Guidance for Catheterization Skill Transfer

Abstract

With growing demands in endovascular intervention, priority is expected to be given to the catheter manipulation skill training. For novices, it is inevitable to face the dilemma that progress is achieved slowly when they practice repeatedly. Thus, exploring underlying behavior patterns from experts’ demonstrations and transferring manipulation skills to novice are of significance when designing catheterization training systems. To address such issues, a deep reinforcement learning-based VR training system has been presented, which aims to exploit expert’s experience-related skills to correct novices’ actions via the 2-DOF haptic guidance. The training system mainly consists of three parts: a force sensors’ integrated catheter manipulator, a VR simulator, and a learning-based model. As experience-related skills are encoded as the correlation between operator–tool and tool–tissue interactions, the deep deterministic policy gradient (DDPG), one of deep reinforcement learning algorithms, is adopted so that expert’s skills can be extracted as the optimal policy model by learning from the catheterization demonstrations. Based on such learning model, the catheter manipulator can provide the intuitive haptic guidance to online correct novices’ catheterization maneuvers both in translational and rotational directions. To evaluate the skill improvement, a five-day training session has been carried out and the results reveal that prominent progress has been made with respect to safety operation and task proficiency. Moreover, substantial evidences support the finding that the newly developed training system is qualified to transfer experience-related skills from expert to novice because trainees are capable of making wise decisions independently without any haptic cues after training sessions.