Monocular camera-based online sensor-to-segment calibration for upper body pose estimation

Abstract

Robot-assisted rehabilitation training usually requires body pose estimation for either evaluation or robot control. Wearable sensors such as inertial measurement units (IMUs) have been extensively studied as affordable solutions. However, a prerequisite is to calibrate the relative transform between the sensor frame and segment anatomical frame, which is usually complex and time-consuming. In this paper, a visual-inertial fusion approach is proposed where a monocular camera is employed to overcome the drawbacks of IMUs and simplify the calibration process. The locations of IMUs are detected from vision using ArUco markers attached to them and the sensor-to-segment calibration is achieved using additional skeletal keypoint locations detected using the MediaPipe framework during arbitrary movements. Experiments were conducted to evaluate the accuracy of the proposed calibration method and the root mean square error of the shoulder and elbow joint angles are 5.13–7.86° with correlation coefficients above 0.96 in comparison to ground truth joint angles from the optical motion capture system. The proposed method achieves similar accuracy to traditional predefined poses-based calibration methods but simplifies the procedure, making it suitable for patients who cannot perform poses accurately in rehabilitation.