Seamless Human-robot Interaction Research Articles

CrowdBot: An Open-Environment Robot Management System for On-Campus Services

In contemporary campus environments, the provision of timely and efficient services is increasingly challenging due to limitations in accessibility and the complexity and openness of the environment. Existing service robots, while operational, often struggle with adaptability and dynamic task management, leading to inefficiencies. To overcome these limitations, we introduce CrowdBot, a robot management system that enhances service in campus environments. Our system leverages a hierarchical reinforcement learning-based cloud-edge hybrid scheduling framework (REDIS), for efficient online streaming task assignment and dynamic action scheduling. To verify the REDIS framework, we have developed a digital twin simulation platform, which integrates large language models and hot-swapping technology. This facilitates seamless human-robot interaction, efficient task allocation, and cost-effective execution through the reuse of robot equipment. Our comprehensive simulations corroborate the system's remarkable efficacy, demonstrating significant improvements with a 24.46% reduction in task completion times, a 9.37% decrease in travel distances, and up to a 3% savings in power usage. Additionally, the system achieves a 7.95% increase in the number of tasks completed and a 9.49% reduction in response time. Real-world case studies further affirm CrowdBot's capability to adeptly execute tasks and judiciously recycle resources, thereby offering a smart and viable solution for the streamlined management of campus services.

Modelling and Measuring Trust in Human–Robot Collaboration

Recognizing trust as a pivotal element for success within Human–Robot Collaboration (HRC) environments, this article examines its nature, exploring the different dimensions of trust, analysing the factors affecting each of them, and proposing alternatives for trust measurement. To do so, we designed an experimental procedure involving 50 participants interacting with a modified ‘Inspector game’ while we monitored their brain, electrodermal, respiratory, and ocular activities. This procedure allowed us to map dispositional (static individual baseline) and learned (dynamic, based on prior interactions) dimensions of trust, considering both demographic and psychophysiological aspects. Our findings challenge traditional assumptions regarding the dispositional dimension of trust and establish clear evidence that the first interactions are critical for the trust-building process and the temporal evolution of trust. By identifying more significant psychophysiological features for trust detection and underscoring the importance of individualized trust assessment, this research contributes to understanding the nature of trust in HRC. Such insights are crucial for enabling more seamless human–robot interaction in collaborative environments.

Innovative Personal Assistance: Speech Recognition and NLP-Driven Robot Prototype

This paper presents the development and evaluation of a personal assistant robot prototype with advanced speech recognition and natural language processing (NLP) capabilities. Powered by a Raspberry Pi microprocessor, it is the core component of the robot's hardware. It is designed to receive commands and promptly respond by performing the requested actions, utilizing integrated speech recognition and NLP technologies. The prototype aims to enhance meeting efficiency and productivity through audio-to-text conversion and high-quality image capture. Results show excellent performance, with accuracy rates of 100% in Indonesian and 99% in English. The efficient processing speed, averaging 9.07 seconds per minute in Indonesian and 15.3 seconds per minute in English, further enhances the robot's functionality. Additionally, integrating a high-resolution webcam enables high-quality image capture at 1280 x 720 pixels. Real-time integration with Google Drive ensures secure storage and seamless data management. The findings highlight the prototype's effectiveness in facilitating smooth interactions and effective communication, leveraging NLP for intelligent language understanding. Integrating NLP-based speech recognition, visual documentation, and data transfer provides a comprehensive platform for managing audio, text, and image data. The personal assistant robot prototype presented in this research represents a significant advancement in human-robot interaction, particularly in meeting and collaborative work settings. Further refinements in NLP can enhance efficiency and foster seamless human-robot interaction experiences.

Snapture—a Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition

As robots are expected to get more involved in people’s everyday lives, frameworks that enable intuitive user interfaces are in demand. Hand gesture recognition systems provide a natural way of communication and, thus, are an integral part of seamless human-robot interaction (HRI). Recent years have witnessed an immense evolution of computational models powered by deep learning. However, state-of-the-art models fall short of expanding across different gesture domains, such as emblems and co-speech. In this paper, we propose a novel hybrid hand gesture recognition system. Our Snapture architecture enables learning both static and dynamic gestures: by capturing a so-called snapshot of the gesture performance at its peak, we integrate the hand pose and the dynamic movement. Moreover, we present a method for analyzing the motion profile of a gesture to uncover its dynamic characteristics, which allows regulating a static channel based on the amount of motion. Our evaluation demonstrates the superiority of our approach on two gesture benchmarks compared to a state-of-the-art CNNLSTM baseline. Our analysis on a gesture class basis unveils the potential of our Snapture architecture for performance improvements using RGB data. Thanks to its modular implementation, our framework allows the integration of other multimodal data, like facial expressions and head tracking, which are essential cues in HRI scenarios, into one architecture. Thus, our work contributes both to integrative gesture recognition research and machine learning applications for non-verbal communication with robots.

A Human-Centered and Adaptive Robotic System Using Deep Learning and Adaptive Predictive Controllers

The rise of single-person households coupled with a drop in social interaction due to the coronavirus disease 2019 (COVID-19) pandemic is triggering a loneliness pandemic. This social issue is producing mental health conditions (e.g., depression and stress) not only in the elderly population but also in young adults. In this context, social robots emerge as human-centered robotics technology that can potentially reduce mental health distress produced by social isolation. However, current robotics systems still do not reach a sufficient communication level to produce an effective coexistence with humans. This paper contributes to the ongoing efforts to produce a more seamless human-robot interaction. For this, we present a novel cognitive architecture that uses (i) deep learning methods for mood recognition from visual and voice modalities, (ii) personality and mood models for adaptation of robot behaviors, and (iii) adaptive generalized predictive controllers (AGPC) to produce suitable robot reactions. Experimental results indicate that our proposed system influenced people’s moods, potentially reducing stress levels during human-robot interaction.

Deep learning-based approaches for human motion decoding in smart walkers for rehabilitation

Gait disabilities are among the most frequent impairments worldwide. Their treatment increasingly relies on rehabilitation therapies, in which smart walkers are being introduced to empower the user’s recovery state and autonomy, while reducing the clinicians effort. For that, these should be able to decode human motion and needs, as early as possible. Current walkers decode motion intention using information gathered from wearable or embedded sensors, namely inertial units, force sensors, hall sensors, and lasers, whose main limitations imply an expensive solution or hinder the perception of human movement. Smart walkers commonly lack an advanced and seamless human–robot interaction, which intuitively and promptly understands human motions. A contactless approach is proposed in this work, addressing human motion decoding as an early action recognition/detection problematic, using RGB-D cameras. We studied different deep learning-based algorithms, organised in three different approaches, to process lower body RGB-D video sequences, recorded from an embedded camera of a smart walker, and classify them into 4 classes (stop, walk, turn right/left). A custom dataset involving 15 healthy participants walking with the walker device was acquired and prepared, resulting in 28800 balanced RGB-D frames, to train and evaluate the deep learning networks. The best results were attained by a convolutional neural network with a channel-wise attention mechanism, reaching accuracy values of 99.61% and above 93%, for offline early detection/recognition and trial simulations, respectively. Following the hypothesis that human lower body features encode prominent information, fostering a more robust prediction towards real-time applications, the algorithm focus was also quantitatively evaluated using Dice metric, leading to values slightly higher than 30%. Promising results were attained for early action detection as a human motion decoding strategy, with enhancements in the focus of the proposed architectures.

Augmented Reality for Human-Robot Collaboration and Cooperation in Industrial Applications: A Systematic Literature Review.

With the continuously growing usage of collaborative robots in industry, the need for achieving a seamless human–robot interaction has also increased, considering that it is a key factor towards reaching a more flexible, effective, and efficient production line. As a prominent and prospective tool to support the human operator to understand and interact with robots, Augmented Reality (AR) has been employed in numerous human–robot collaborative and cooperative industrial applications. Therefore, this systematic literature review critically appraises 32 papers’ published between 2016 and 2021 to identify the main employed AR technologies, outline the current state of the art of augmented reality for human–robot collaboration and cooperation, and point out future developments for this research field. Results suggest that this is still an expanding research field, especially with the advent of recent advancements regarding head-mounted displays (HMDs). Moreover, projector-based and HMDs developed approaches are showing promising positive influences over operator-related aspects such as performance, task awareness, and safety feeling, even though HMDs need further maturation in ergonomic aspects. Further research should focus on large-scale assessment of the proposed solutions in industrial environments, involving the solution’s target audience, and on establishing standards and guidelines for developing AR assistance systems.

Modeling and Fabrication of a Piezoelectric Artificial Cochlea Electrode Array With Longitudinal Coupling

In this study, inspired from human cochlea, we propose an artificial basilar membrane structure based on an array of gold electrodes microfabricated along the opposite surfaces of a piezoelectric continuum. The proposed piezoelectric continuum features an isosceles trapezoidal geometry and is situated atop an elastomer matrix with embedded electrode grooves and a duct. At first, a detailed analytical model is presented, which formulates the continuum’s planar deflection equation, governing its elastodynamic behavior. Subsequently, in order to verify the analytical outcomes, a multiphysics finite-element analysis is performed in COMSOL®. Both analytical and numerical models demonstrate desired frequency selectivity akin to the capability of a biological basilar membrane. Subsequently, a proof-of-concept model of the proposed structure is custom-fabricated through standard photo- and soft lithography techniques using Polyvinylidene Difluoride (PVDF), gold, and Polydimethylsiloxane (PDMS) materials. Finally, the fabricated artificial basilar membrane that is intended to be used as a broad band sensor, and artificial cochlea for seamless human-robot interaction is experimentally characterized using a custom-built acoustic exciter, and its effective performance is validated within the frequency range of 3 kHz to 8 kHz.

An Implementation of Seamless Human-Robot Interaction for Telerobotics

Achieving human-robot cooperation in a telerobotics system is the focus of this paper. The implementation of human-robot systems can be extremely challenging when the robot is not directly controlled by the human. The interaction mode, depending on the task context, can be continuous manual, semi-autonomous or autonomous. To address the Human-Robot Interaction (HRI) issues in such a system, a concept of seamless HRI is introduced. Seamless HRI implies flexibility in human control in interacting with a robot in different situations, and the adaptability of the robot autonomy in response to the human control. The main idea is to design a telerobotics system that allows a shift from manual to autonomous operation, dynamically, via different human-robot roles and relationships. These roles are Master-Slave, Supervisor-Subordinate, Partner-Partner, Teacher-Learner and Full Autonomous mode by the robot. This paper presents the theoretical foundations and the requirements for seamless HRI. An implementation of the concept and an experimental evaluation is presented.