Vision-based human wrist localization and with Kalman-filter backed stabilization for Bilateral Teleoperation of Robotic Arm

Abstract

The development of advanced robotic systems that can be operated effortlessly by humans is crucial for expanding the capabilities of human-robot interaction and collaboration. In this work, we propose a system that enables real-time and non-invasive tracking of the human wrist's position in three-dimensional space to facilitate an intuitive control of a remotely placed robotic arm, as an improvement towards efficient control mechanism. We leverage color segmentation based human wrist tracking in the viewing plane augmented with a stereo depth estimation for synthesising the localized wrist in the Cartesian space. The centroid of the segmented image is determined to generate the precise coordinates. To remove the variance in the estimated centroid, the trajectory of the localized points is modelled by Kalman filtering approach. Finally, the coordinates are transformed using Inverse Imaging Model and transmitted to an edge device (close to the robot), which then replicates the motion at the robot end using inverse kinematics. The results show a high degree of accuracy in tracking the wrist's movements, enabling precise and natural control over the robotic arm's motion. The system's adaptability to various environments and user populations is validated through tests with diverse subjects. With its high accuracy, real-time performance, the proposed approach holds great potential to enhance the efficiency and safety of human-robot collaborative applications across multiple domains.