A speech-enabled virtual assistant for efficient human–robot interaction in industrial environments

Abstract

This paper presents a natural language-enabled virtual assistant (VA), named Max, developed to support flexible and scalable human–robot interactions (HRI) with industrial robots. Regardless of the numerous natural language interfaces already proposed for intuitive HRI on the industrial shop floor, most of those interfaces remain tightly bound with a specific robotic system. Besides, the lack of a natural and efficient human–robot communication protocol hinders the user experience. Therefore three key elements characterize the proposed framework. First, a Client–Server style architecture is introduced so Max can provide a centralized solution for managing and controlling various types of robots deployed on the shop floor. Second, inspired by human–human communication, two conversation strategies, lexical-semantic and general diversion strategies, are used to guide Max’s response generation. These conversation strategies were embedded to improve the operator’s engagement with the manufacturing tasks. Third, we fine-tuned the state-of-the-art (SOTA) pre-trained model, Bidirectional Encoder Representations from Transformers (BERT), to support a highly accurate prediction of requested intents from the operator and robot services. Multiple experiments were conducted using the latest iteration of our autonomous industrial mobile manipulator, “Little Helper (LH)”, to validate Max’s performance in a real manufacturing environment.