Simple AR Method for Rehabilitation Support System Based on 3D Pose Estimation

Abstract

Studies have been conducted on the application of Augmented Reality to support rehabilitation of motor function recovery. The goal of these studies is to facilitate functional recovery training through patient interaction with virtual objects generated by AR. Many of them use special devices such as depth sensors to superimpose virtual objects at appropriate positions in images, but a simple method that does not require such a device is desired. In order to realize superimposition using only a personal computer (PC) with a camera, this study utilizes a deep neural network that estimates the 3-dimensional (3D) coordinates of keypoints, such as human joints, from camera images. Specifically, a coordinate transformation matrix for superimposition is calculated from the 3D coordinates of keypoints. In order to clarify the effectiveness of this method, we conducted an experiment to evaluate the superimposition accuracy. The results show that the accuracy was highest in the space near the keypoints that had been used to compute the coordinate transformation matrix, and the accuracy was even higher when the number of keypoints was small. This indicates that this method is more suitable for localized training such as hand rehabilitation than for whole-body training. Since this method can be used only with a PC with a camera, it is expected to be widely used for rehabilitation support.