Multimodal Robot Research Articles

Multi-Modal and Adaptive Robot Control through Hierarchical Quadratic Programming

This paper proposes a novel Hierarchical Quadratic Programming (HQP)-based framework that enables multi-tasking control under multiple Human-Robot Interaction (HRI) scenarios. The proposed controllers’ formulations are inspired by real-world contact-rich scenarios, which currently constitute one of the main limitations in terms of widespread practical deployment. Indeed, HRI can occur through different modalities, based on human’s needs. The objective is to create a unique framework for various types of possible interactions, avoiding the necessity of switching between different control architectures, which requires dealing with discontinuities. To achieve this, we firstly propose a HQP-based hybrid Cartesian/joint space impedance control formulation. Based on the robot’s dynamics, this controller enables an adaptive compliance behaviour, while achieving hierarchical motion control. This is validated through a series of experiments that show the accuracy of trajectory tracking, which remains in the order of 10mm during fast motions thanks to the addition of the robot dynamics. Besides, the hybrid compliance behaviour allows to deviate from such accuracy when an interaction is present. We then consider the case in which the human needs to move the robot directly, by proposing a hybrid admittance/impedance controller, that is again based on a HQP formulation and provides inherent softening when conflicting tasks are present, or in close-to-limit and near-singular configurationsa. This is validated through several experiments in which the human easily moves the robot in the workspace via direct physical interaction. Next, we formulate an additional hierarchy that enables force control and allows to maintain a specific interaction force at the end effector. We then extend this to simultaneous force and trajectory tracking. Overall, we obtain a multi-purpose HQP-based control framework, that seamlessly switchwes between interaction modes, enabling multiple hierarchical behaviours, and covering a wide spectrum of interaction types, essential for synergistic HRI.

Magnetic hexagram micro soft robot

Abstract. This research focuses on the development of a hexagram-shaped multimodal magnetic soft robot for biomedical applications. This micro-robot utilizes principles of soft robotics and magnetic control mechanisms to achieve precise manipulation and maneuverability within biological environments at the micrometer to millimeter scale. Its advantages include precise navigation through physiological fluids via magnetic fields, enabling targeted drug delivery, diagnostic imaging enhancement, and minimally invasive surgical interventions. The hexagram shape ensures stable and controlled movements, overcoming instability seen in alternative robotic systems. However, these magnetic robots also face challenges such as size constraints, navigating turbulent flows and anatomical obstacles, sustaining operations with an efficient energy source, and biocompatibility concerns.

A multimodal educational robots driven via dynamic attention.

With the development of artificial intelligence and robotics technology, the application of educational robots in teaching is becoming increasingly popular. However, effectively evaluating and optimizing multimodal educational robots remains a challenge. This study introduces Res-ALBEF, a multimodal educational robot framework driven by dynamic attention. Res-ALBEF enhances the ALBEF (Align Before Fuse) method by incorporating residual connections to align visual and textual data more effectively before fusion. In addition, the model integrates a VGG19-based convolutional network for image feature extraction and utilizes a dynamic attention mechanism to dynamically focus on relevant parts of multimodal inputs. Our model was trained using a diverse dataset consisting of 50,000 multimodal educational instances, covering a variety of subjects and instructional content. The evaluation on an independent validation set of 10,000 samples demonstrated significant performance improvements: the model achieved an overall accuracy of 97.38% in educational content recognition. These results highlight the model's ability to improve alignment and fusion of multimodal information, making it a robust solution for multimodal educational robots.

A multimodal continuum robot constructed using three stable state characteristics of Kresling origami

Abstract Inspired by the Kresling origami structure and its three stable state characteristics, a novel multimodal continuum robot (MCR) has been proposed in this paper, which achieve its own movement through gas-driven and actively elastic deformed Kresling origami. Air bags are utilized to auxiliary drive in this paper, which make the specific plane of origami unit transform from first stable state to third stable state. The high stiffness of third stable state endows the origami unit better carrying capacity and more motion characteristics, the driving force of the central chamber and selective expansion of air bags enable the multimodal motion of MCR. The theoretical model of MCR and three stable state characteristics of Kresling origami are established to predict and control the behavior of actuator. The experimental results of MCR prototype demonstrate the high carrying capacity of proposed origami unit, the feasibility and high accuracy of multimodal motion. The designing concept of this paper improves the ability of continuum robots to perform complex tasks and promotes the development of origami structures in the field of robotics.

Adapting to My User, Engaging with My Robot: An Adaptive Affective Architecture for a Social Assistive Robot

Affective feedback from social robots is a useful technique for communicating to people whether they are interacting “well” with the robot or not. However, some users, such as people with physical or cognitive difficulties, may not be able to interact in all the desired ways. In these cases, affective feedback from the robot could be excessively negative—an “unhappy” robot, leading to an unrewarding experience for the user. This paper presents a motivation-based architecture for an autonomous multimodal social robot, that incorporates an affective feedback mechanism which generates an affective state by combining the internal needs of the robot and the social interaction quality. The balance between these two factors can dynamically change, allowing the robot to adapt its affective feedback to the user’s interaction style and capabilities. We have implemented this architecture in a simulation and in a MiRo social robot, and report experiments examining the behavior of the system in interactions with different experimental user profiles. The results show that the adaptive mechanism allows the robot to change its affective feedback to give more positive encouragement to users than in non-adaptive cases.

RL-CWtrans Net: multimodal swimming coaching driven via robot vision.

In swimming, the posture and technique of athletes are crucial for improving performance. However, traditional swimming coaches often struggle to capture and analyze athletes' movements in real-time, which limits the effectiveness of coaching. Therefore, this paper proposes RL-CWtrans Net: a robot vision-driven multimodal swimming training system that provides precise and real-time guidance and feedback to swimmers. The system utilizes the Swin-Transformer as a computer vision model to effectively extract the motion and posture features of swimmers. Additionally, with the help of the CLIP model, the system can understand natural language instructions and descriptions related to swimming. By integrating visual and textual features, the system achieves a more comprehensive and accurate information representation. Finally, by employing reinforcement learning to train an intelligent agent, the system can provide personalized guidance and feedback based on multimodal inputs. Experimental results demonstrate significant advancements in accuracy and practicality for this multimodal robot swimming coaching system. The system is capable of capturing real-time movements and providing immediate feedback, thereby enhancing the effectiveness of swimming instruction. This technology holds promise.

The Multi-Modal Robot Perception, Language Information, and Environment Prediction Model Based on Deep Learning

This paper presents a deep learning-based model to enhance robot perception and prediction in complex environments. The model utilizes CNN, LSTM, and attention mechanisms to address multimodal perception and environment prediction tasks. The introduction provides background on the importance of robots in various fields and emphasizes the need for multimodal perception. The methodology section explains the working principles of CNN, LSTM, and attention mechanisms. The experimental results demonstrate significant improvements in robot perception and prediction, with low errors and high accuracy. The proposed model supports robot applications in complex environments such as autonomous navigation and human-robot interaction. It effectively handles multimodal data and enhances robot perception and prediction capabilities. The experimental results validate the model's effectiveness, providing important support for robot decision-making in complex environments.

Multimodal locomotion ultra-thin soft robots for exploration of narrow spaces

From power plants on land to bridges over the sea, safety-critical built environments require periodic inspections for detecting issues to avoid functional discontinuities of these installations. However, navigation paths in these environments are usually challenging as they often contain difficult-to-access spaces (near-millimetre and submillimetre-high gaps) and multiple domains (solid, liquid and even aerial). In this paper, we address these challenges by developing a class of Thin Soft Robots (TS-Robot: thickness, 1.7 mm) that can access narrow spaces and perform cross-domain multimodal locomotion. We adopted a dual-actuation sandwich structure with a tuneable Poisson’s ratio tensioning mechanism for developing the TS-Robots driven by dielectric elastomers, providing them with two types of gaits (linear and undulating), remarkable output force ( ~ 41 times their weight) and speed (1.16 times Body Length/s and 13.06 times Body Thickness/s). Here, we demonstrated that TS-Robots can crawl, climb, swim and collaborate for transitioning between domains in environments with narrow entries.

Intelligent Rock-Climbing Robot Capable of Multimodal Locomotion and Hybrid Bioinspired Attachment.

Rock-climbing robots have significant potential in fieldwork and planetary exploration. However, they currently face limitations such as a lack of stability and adaptability on extreme terrains, slow locomotion, and single functionality. This study introduces a novel multimodal and adaptive rock-climbing robot (MARCBot), which addresses these limitations through spiny grippers that draw inspiration from morpho-functionalities observed in beetles, arboreal birds, and hoofed animals. This hybrid bioinspired design enables high attachment strength, passive adaptability to different terrains, and quick attachment on rock surfaces. The multimodal functionality of the gripper allows for attachment during climbing and support during walking. A novel control strategy using dynamics and quadratic programming (QP) optimizes attachment wrench distribution, reducing cost-of-transport by 20.03% and 6.05% compared to closed-loop inverse kinematic (CLIK) and virtual model control (VMC) methods, respectively. MARCBot achieved climbing speeds of 0.15mmin-1 on a vertical discrete rock surface under gravity and trotting speeds of up to 0.21ms-1 on various complex terrains. It is the first robot capable of climbing on rock surfaces and trotting in complex terrains without the need for switching end-effectors. This study highlights significant advancements in climbing and multimodal locomotion for robots in extreme environments.

MTFR: An universal multimodal fusion method through Modality Transfer and Fusion Refinement

Multimodal learning has made great achievements in recent years. However, most of the existing modal fusion methods capture cross-modal correlations in a single stage and are designed for specific tasks, with poor generalizability and portability. In this work, we propose the Modality Transfer and Fusion Refinement method (MTFR), which contains two feature-level fusion modules and can be used for several applications. Specifically, the Modality Transfer greatly shortens the distance between modalities through transferring non-text information into text-like representations. Fusion Refinement corrects and complements the previous fusion process to make the model focus more on task-specific knowledge by introducing uni-modal processing. To illustrate the validity of MTFR, we design three experiments for sarcasm detection, sentiment analysis, and offensiveness detection. The experimental results validate that our method achieves competitive performance in all three applications and outperforms both task-specific models and multimodal fusion baselines, which demonstrate the good generalizability and robustness of our method. Thanks to the promising performance, the method will facilitate the development of multimodal fusion and can promote multimodal applications in practice such as social sentiment analysis and multimodal conversational robots.

Hardware Conceptual Design of Tsunhan-14 Multimodal Robot Observer in Support of TNI Tasks

The development of technology in the military field has changed from time to time. This dynamic must be taken seriously because they are a challenge, and pose a potential threat to the sovereignty of the Unitary State of the Republic of Indonesia (NKRI), As well as the trend of the technology-based warfare today. So it has become a necessity for the Indonesian National Army (TNI) to accept it. One of the popular technologies today, war robt technology(warbo), and including multimodal robots with a variety of basic missions. Furthermore, developing a multimodal robot concept design that can fulfil TNI surveillance needs. The methodology used is in the form of interview, observations, comparison, and literature studies by applying system engineering theory and life cycle models. This research has successfully formulated user requirement objectivies which are then elaborated into robot design concepts including characteristic design, and architecture design of multimodal monitoring robots. The hardware on the robot is design differently from others because it has the ability to fly and walk on the ground. The robot is also conceptualised to have advantages in terms of durability with agility, silence, and operate with various terrain conditions. Equipped with sensors to see, hear, measure ambient temperature, and read surrounding conditions using camera sensors, where the robot moves autonomously equipped with GPS and Lidar can send data in the form of audio visual directly to the Ground Control System (command centre). The concept design of the multimodal monitoring robot called Tsunhan-14 is expected to be the basis for determining technical requirement and further developed into a national innovation that is utilized as part of the TNI’s war strategy.

Tumro: A Tunable Multimodal Wheeled Jumping Robot Based on the Bionic Mechanism of Jumping Beetles

The implementation of multimodal motion ensures the stable operation in complex terrain environments, thus providing an effective guarantee for system performance. The crawling‐jumping robot has the ability to navigate in various road conditions utilizing different modes of movement. However, the mobility of the current multimodal jumping robots remains somewhat constrained by their jumping capability and the recovery time after each jump. Drawing inspiration from the energy‐storage jumping mechanism of jumping beetles, a tuneable multimodal jumping robot (Tumro) capable of executing multimodal movements including wheeled locomotion and ground‐based jumping, which can achieve a jump height of up to 3 m and swiftly recover its wheeled crawling state without requiring posture correction post‐jump, is presented. Through a specific structural design, the robot can storage energy and switch motions to jump in the desired direction based on the preset angle according to actual demand. The jumping process is thoroughly analyzed, and the kinematics and dynamics models are derived in meticulous detail. In addition, the performance of the robot is comprehensively assessed from aspects of wheel action versus vertical jump capability, power consumption, and endurance across various motion modes. The simulation scene experiment demonstrates the robot's exceptional jumping capability and efficient wheeled mobility.

A Multi-Modal Tailless Flapping-Wing Robot Capable of Flying, Crawling, Self-Righting and Horizontal Take-Off

A Multi-Modal Tailless Flapping-Wing Robot Capable of Flying, Crawling, Self-Righting and Horizontal Take-Off

L-NORM: Learning and Network Orchestration at the Edge for Robot Connectivity and Mobility in Factory Floor Environments

Robotic factory floors will revolutionize the future of manufacturing and the service industry by automating tasks. However, to fully supplement human effort, these robots will need low-latency, reliable connectivity throughout the work zone through links established by wireless access points (APs). This will allow the robot to assuredly respond to programming directives that rely on the real-time relaying of robot-generated sensor data to the Mobile Edge Computing (MEC) server. In this paper, we propose L-NORM, a multi-AP and multi-robot coordination framework, as a multi-tiered solution for such autonomous edge networks. First, multi-robot motion planning through reinforcement learning occurs at the MEC, using as input multi-modal robot sensor data. Second, multi-AP resource orchestration is performed using another reinforcement learning-based method that maps a subset of available APs to each robot toward meeting their sensor data delivery requirements. Furthermore, we suggest diversity combination of uplink channels with the 802.11ax scheduled access mode that will (i) support high reliability of multi-robot uplink sensor packets and (ii) enable multi-AP coordination, for optimized resource utilization. Through extensive simulation studies, we show that the probability of robot deviation to remain within 0.5 m from its optimal path, is 19% more in L-NORM compared to classical 802.11ax based edge network solution, considering <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 1 MB of sensor data per robot.

Design and multimodal locomotion plan of a hexapod robot with improved knee joints

AbstractHexapod robots represent complex and highly redundant robotic systems capable of handling various tasks in extreme environments, such as planetary exploration and disaster response. To enhance the terrain adaptability and maneuverability of hexapod robots, we present a novel multimodal hexapod robot in this research. Our multimodal hexapod robot design incorporates knee joints with an improved multiparallel quadrilateral transmission mechanism, addressing singularity issues commonly encountered in legged robots. This enhancement not only improves the mechanical transmission characteristics of the knee joints but also increases the workspace of a single leg to achieve leg‐arm reuse functionality. In the presence of flat and structured terrain, the robot seamlessly switches to a wheeled locomotion mode, enabling swift traversal. Conversely, when faced with rough and unstructured terrain, it transitions into a legged mode, employing adaptive gait stability to navigate effectively. A novel operational mode in which the hexapod robot utilizes four legs for body support, while the remaining two legs are repurposed as arms is proposed. This innovative approach empowers the hexapod robot to perform dual‐arm manipulation without the need for additional degrees of freedom for the manipulator. To validate the effectiveness of our designed hexapod robot and multimodal motion planning algorithm, comprehensive experimental testing has been conducted, demonstrating the practicality and versatility of our system.

Interoperable communication for autonomous mobile robots

The deployment of teams of autonomous mobile robots provides potential by leveraging the strengths of individual robots and compensating for their disadvantages. This makes them particularly well suited to carrying out complex scenarios for logistics, environmental monitoring or search and rescue. However, the success of such teams relies on achieving seamless interoperability among differing robots. In the area of the Internet of Robotic Things (IoRT), addressing this interoperability challenge becomes paramount. Therefore, an interoperable communication concept is introduced that considers interoperability based on semantic, syntactic, and technical clusters. It is designed to meet the specific requirements of multimodal autonomous mobile robots. Based on legal considerations, this method enables the integration and collaboration of different autonomous mobile robots in real-life applications. A logistics scenario involving multiple autonomous mobile robots illustrates the applicability of this interoperable communication concept.

YOLOv8-PoseBoost: Advancements in Multimodal Robot Pose Keypoint Detection

In the field of multimodal robotics, achieving comprehensive and accurate perception of the surrounding environment is a highly sought-after objective. However, current methods still have limitations in motion keypoint detection, especially in scenarios involving small target detection and complex scenes. To address these challenges, we propose an innovative approach known as YOLOv8-PoseBoost. This method introduces the Channel Attention Module (CBAM) to enhance the network’s focus on small targets, thereby increasing sensitivity to small target individuals. Additionally, we employ multiple scale detection heads, enabling the algorithm to comprehensively detect individuals of varying sizes in images. The incorporation of cross-level connectivity channels further enhances the fusion of features between shallow and deep networks, reducing the rate of missed detections for small target individuals. We also introduce a Scale Invariant Intersection over Union (SIoU) redefined bounding box regression localization loss function, which accelerates model training convergence and improves detection accuracy. Through a series of experiments, we validate YOLOv8-PoseBoost’s outstanding performance in motion keypoint detection for small targets and complex scenes. This innovative approach provides an effective solution for enhancing the perception and execution capabilities of multimodal robots. It has the potential to drive the development of multimodal robots across various application domains, holding both theoretical and practical significance.

Multimodal audio-visual robot fusing 3D CNN and CRNN for player behavior recognition and prediction in basketball matches.

Intelligent robots play a crucial role in enhancing efficiency, reducing costs, and improving safety in the logistics industry. However, traditional path planning methods often struggle to adapt to dynamic environments, leading to issues such as collisions and conflicts. This study aims to address the challenges of path planning and control for logistics robots in complex environments. The proposed method integrates information from different perception modalities to achieve more accurate path planning and obstacle avoidance control, thereby enhancing the autonomy and reliability of logistics robots. Firstly, a 3D convolutional neural network (CNN) is employed to learn the feature representation of objects in the environment for object recognition. Next, long short-term memory (LSTM) is used to model spatio-temporal features and predict the behavior and trajectory of dynamic obstacles. This enables the robot to accurately predict the future position of obstacles in complex environments, reducing collision risks. Finally, the Dijkstra algorithm is applied for path planning and control decisions to ensure the robot selects the optimal path in various scenarios. Experimental results demonstrate the effectiveness of the proposed method in terms of path planning accuracy and obstacle avoidance performance. The method outperforms traditional approaches, showing significant improvements in both aspects. The intelligent path planning and control scheme presented in this paper enhances the practicality of logistics robots in complex environments, thereby promoting efficiency and safety in the logistics industry.

Multimodal Robot Programming Interface Based On RGB-D Perception and Neural Scene Understanding Modules

In this paper, we propose a system for natural and intuitive interaction with the robot. Its purpose is to allow a person with no specialized knowledge and no training in robot programming to program a robotic arm.We utilize data from the RGB-D camera to segment the scene and detect objects. We also estimate the configuration of the operator's hand and the position of the visual marker to determine the intentions of the operator and the actions of the robot. To this end, we utilize trained neural networks and operations on the input point clouds. Also, voice commands are used to define or trigger the execution of the motion. Finally, we performed a set of experiments to show the properties of the proposed system.

Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception.

The field of multimodal robotic musical performing arts has garnered significant interest due to its innovative potential. Conventional robots face limitations in understanding emotions and artistic expression in musical performances. Therefore, this paper explores the application of multimodal robots that integrate visual and auditory perception to enhance the quality and artistic expression in music performance. Our approach involves integrating GRU (Gated Recurrent Unit) and GoogLeNet models for sentiment analysis. The GRU model processes audio data and captures the temporal dynamics of musical elements, including long-term dependencies, to extract emotional information. The GoogLeNet model excels in image processing, extracting complex visual details and aesthetic features. This synergy deepens the understanding of musical and visual elements, aiming to produce more emotionally resonant and interactive robot performances. Experimental results demonstrate the effectiveness of our approach, showing significant improvements in music performance by multimodal robots. These robots, equipped with our method, deliver high-quality, artistic performances that effectively evoke emotional engagement from the audience. Multimodal robots that merge audio-visual perception in music performance enrich the art form and offer diverse human-machine interactions. This research demonstrates the potential of multimodal robots in music performance, promoting the integration of technology and art. It opens new realms in performing arts and human-robot interactions, offering a unique and innovative experience. Our findings provide valuable insights for the development of multimodal robots in the performing arts sector.