A collaborative telerobotics network framework with hand gesture interface and conflict prevention

Abstract

Hand gesture control is an efficient modality for telerobot control because of gesture expressiveness and naturalness. This paper discusses a collaborative cybernetic system, where telerobots are controlled simultaneously by a group of distributed operators over the network to accomplish a task. Computer vision algorithms for hand gesture recognition are introduced to facilitate the human–robot interface. The gestures are converted into commands that are delivered to robots for dexterous task completion. Commands from multiple operators are aggregated by a collaboration protocol into a single control stream. The aggregation is updated according to operators’ performance. A distributed conflict and error detection–prediction network is designed and applied to a case study of collaborative control for a robotic nuclear decommissioning task. Operators use hand gestures to command telerobots to disassemble facilities in a contaminated area. The hypothesis is tested that collaborative control is more effective and less susceptible to conflicts/errors than the standard single-operator control. During collaboration, operators performed gesture commands simultaneously to control a set of robots. The system can reliably recognise operators’ hands with a 96% accuracy in cluttered backgrounds. Collaboration between expert and novice operators can reduce the time to complete a multi-step task by 45% on average.