Gaze-aware hand gesture recognition for intelligent construction

Abstract

The advances in construction robotics in recent decades has been a powerful driver of construction automation. User-friendly interfaces to support human–robot work collaboration are critical for increasing adoption of construction robots. Among different interfaces, hand gesture is an effective and reliable interaction cue in the noisy construction environment. This paper proposes a novel human gaze-aware hand gesture recognition framework as a human–robot interface for intelligent construction. The proposed framework relies on an eye tracker to visually detect and track robotic machines in the first-person view. Then, the machine-of-interest is determined based on the bounding boxes of machines and gaze points. Finally, a hand gesture recognition architecture is incorporated with the machine information for conveying messages to the machine-of-interest. This approach was tested through a framework validation test and achieved precision and recall of 93.8% and 95.0%, respectively. A pilot study was conducted to demonstrate interaction with a robotic excavator and a dump truck, and the results illustrated that the proposed framework could serve as an effective interface for one-to-many human–robot collaborations in construction.