MES: A Helping Elderly Escort Interactive System Based on Reverse Active Integration of Multimodal Intentions

Abstract

Lacking human-computer collaborative capability is one of the key problems commonly faced by the existing escort robots. To this end, a helping elderly escort interactive system based on reverse active integration of multimodal intentions (MES) is proposed in this paper in response to the elderly’s decline in language expression ability, memory, and other abilities. This system can understand the intention of the elderly based on scene perception and three-modal information: speech, gesture, and posture. In detail, the system can extract the interactive intention from the nondeterministic multimodal data input by the elderly and evaluate the trust degree of the extracted intention. The evaluation of intention trust degree is such a process that the system autonomously judges the feasibility of the elderly’s intention and corrects the wrong intention expressed by the elderly due to memory decline by reverse thinking of “find reasons based on the results”; when the intention cannot be extracted, the system will take the initiative to ask the elderly for enhanced information conducive to the intention extraction, so as to quickly and correctly extract the interactive intention of the elderly. This design aims at improving the quality of elderly care, making the interaction between the elderly and the robot more natural, improving the accuracy in intention extraction from fuzzy expression, as well as breaking the traditional “master-slave” human-computer interaction and improving the harmony of human-computer interaction. Further, the implementation principle of the system is detailed, and the system is evaluated in this paper. The evaluation experiment was conducted by a robot Pepper embedded with the system. Through experiments, it is verified that Pepper can quickly and accurately get the real intention of the elderly in the interaction with the elderly. In a challenging environment (such as the unclear expression of the elderly), it can still correctly extract the real intention of the elderly with an accuracy of 97% and can effectively avoid the wrong intention expressed by the elderly. This puts forward a valuable research path for the challenge in human-computer collaborative interaction.