A Multimodal User Interface for an Assistive Robotic Shopping Cart

Dmitry Ryumin,Anton Saveliev,Iosif Mporas,Irina Kipyatkova,Alexandr Axyonov,Ildar Kagirov,Alexey Karpov,Nikita Pavlyuk,Milos Zelezny

doi:10.3390/electronics9122093

Abstract

This paper presents the research and development of the prototype of the assistive mobile information robot (AMIR). The main features of the presented prototype are voice and gesture-based interfaces with Russian speech and sign language recognition and synthesis techniques and a high degree of robot autonomy. AMIR prototype’s aim is to be used as a robotic cart for shopping in grocery stores and/or supermarkets. Among the main topics covered in this paper are the presentation of the interface (three modalities), the single-handed gesture recognition system (based on a collected database of Russian sign language elements), as well as the technical description of the robotic platform (architecture, navigation algorithm). The use of multimodal interfaces, namely the speech and gesture modalities, make human-robot interaction natural and intuitive, as well as sign language recognition allows hearing-impaired people to use this robotic cart. AMIR prototype has promising perspectives for real usage in supermarkets, both due to its assistive capabilities and its multimodal user interface.

Highlights

Assistive robots are robots that help to maintain or enhance the capabilities usually of older persons or people suffering from functional limitations
assistive mobile information robot (AMIR) is an assistive robotic platform with the architecture of the interface being focused on the needs of elders and/or people with hearing impairments
AMIR is the only project that deals with real-time Russian sign language recognition/synthesis and can be effectively used as a service robot, serving the needs of a wide range of supermarket customers

Summary

Introduction

Assistive robots are robots that help to maintain or enhance the capabilities usually of older persons or people suffering from functional limitations. Assistive robots help people with injuries to move and to maintain a good social life status, resulting in psychological and physical well-being. The prime example of this strategy is provided in the EQUAL project [10], aimed at enhancing the quality of life of people suffering from moving disability. The developers conducting the EQUAL project propose a number of steps leading to shopping facilitation. Among the steps are the development of assistive mechanized shopping cart, software, and improved infrastructure supporting people with a moving disability. More broadly, assistive technologies are designed to support or even replace the services provided by caregivers and physicians, reduce the need for regular healthcare services, and make persons who suffer from various dysfunctions more independent in their everyday life.

Objectives

Methods

Results

Conclusion