Multimodal Robot Research Articles

Empowering computing students with proficiency in robotics via situated learning

With the increasing employment of robots in multiple areas such as smart manufacturing and intelligent transportation, both undergraduate and graduate students from computing related majors (e.g., computer science and information technology) demonstrated strong interests in learning robotics technology to broaden their career opportunities. However, instilling computing students with robotics knowledge remains a challenge since most of them have limited pre-training in engineering subjects such as electronics and mechatronics. Therefore, robotics education for computing students demands an immersive real-world learning environment by considering both theories and intensive hands-on projects. Different from traditional textbook-directed robotics learning, in this study, a situated learning-based robotics education pedagogy is proposed for computing students to equip them with robotics expertise and foster their problem-solving skills in real-world human–robot interaction contexts. To create a realistic human–robot collaboration situation, a multi-modal collaborative robot is employed in the classroom-based learning community for the whole semester. Mini-project-based homework and team projects are designed for students to practice their critical thinking and hands-on experiences. The bidirectional-evaluation approach is utilized by the instructor and students to assess the quality of the proposed pedagogy. Practice results and student evaluations suggested that the proposed situated learning-based pedagogy and robotics curriculum provided computing students to learn robotics in an effective way, which was well recognized and accepted by students even most of them were beginners. Future work of this study is also discussed.

A professional slackliner robot.

A multimodal robot demonstrates mobility and balance with the synchronous use of legs and propellers.

Combining Unsupervised Muscle Co-Contraction Estimation With Bio-Feedback Allows Augmented Kinesthetic Teaching

Nowadays, an increasingly diversification of products and production lines would largely benefit from intuitive and multimodal robot teaching strategies. The present article proposes an augmented kinesthetic teaching system, which is based on surface electromyographic (sEMG) measurements from the operator forearm. Specifically, sEMG signals are used for minimal-training unsupervised estimation of forearm's muscles co-contraction level. In this way, also exploiting a vibrotactile bio-feedback, we evaluate the ability of operators in stiffening their hand - during kinesthetic teaching - in order to modulate the estimated level of muscle co-contraction to (i) match target levels and (ii) command the opening/closing of a gripper, i.e. in exploiting their sEMG signals for effective augmented robot kinesthetic teaching tasks. Experiments were carried out involving ten subjects in two different kind of experimental sessions, in order to test both co-contraction modulation abilities, and actual usage of the co-contraction for programming robot functionalities during kinesthetic teaching. The obtained results provide positive outcomes on the intuitiveness and effectiveness of the proposed system and approach, paving the way to a new generation of advanced teaching by demonstration interfaces.

Interactive Force Control Based on Multimodal Robot Skin for Physical Human−Robot Collaboration

This work proposes and realizes a control architecture that can support the deployment of a large‐scale robot skin in a Human‐Robot Collaboration scenario. It is shown, how whole‐body tactile feedback can extend the capabilities of robots during dynamic interactions by providing information about multiple contacts across the robot's surface. Specifically, an uncalibrated skin system is used to implement stable force control while simultaneously handling the multi‐contact interactions of a user. The system formulates control tasks for force control, tactile guidance, collision avoidance, and compliance, and fuses them with a multi‐priority redundancy resolution strategy. The approach is evaluated on an omnidirectional mobile‐manipulator with dual arms covered with robot skin. Results are assessed under dynamic conditions, showing that multi‐modal tactile information enables robust force control while at the same time remaining responsive to a user's interactions.

Multimodal soft jumping robot with self-decision ability

The next generation of soft robots should have self-decision ability to judge reasonable responses adaptively according to external stimulation and without human assistance. However, few of the current soft robots have brains to think independently; rather, they have integrated sensors and actuators, similar to animals having skin and muscles. In this paper, a bistable soft jumping robot with self-decision ability is reported, which is manufactured by 4D printing of composites with continuous fibers. The snap-through triggered by temperature stimulation resulted in the jumping motion of the robot, and its jumping mode is controlled by a programmed continuous fiber pattern. This robot able to take decisions can change from the initial state to the final state via different deforming pathways when snap-through occurs, resulting in multiple jumping motion modes. This soft robot can realize (1) different jumping modes, including forward, rotational, lateral, and overturn movements; (2) self-decision function of motion modes responding to external stimuli; and (3) continuous movement, by repeatedly jumping under periodic temperature stimulation. This research will provide a new scheme for soft robotics to achieve the integration of sensation, actuation, and decision.

Deep Episodic Memory for Verbalization of Robot Experience

The ability to verbalize robot experience in natural language is key for a symbiotic human-robot interaction. While first works approached this problem using template-based verbalization on symbolic episode data only, we explore a novel way in which deep learning methods are used for the creation of an episodic memory from experiences as well as the verbalization of such experience in natural language. To this end, we first collected a complex dataset consisting of more than a thousand multimodal robot episode recordings both from simulation as well as real robot executions, together with representative natural language questions and answers about the robot's past experience. Second, we propose and evaluate an episodic memory verbalization model consisting of a speech encoder and decoder based on the Transformer architecture, combined with an LSTM-based episodic memory auto-encoder, and evaluate the model on simulated and real data from robot execution examples. Our experimental results provide a proof-of-concept for episodic-memory-based verbalization of robot experience.

The Modality Card Deck: Co-Creating Multi-Modal Behavioral Expressions for Social Robots with Older Adults

Robots have been proposed as intelligent technology that can support the independent living and health of older adults. While significant advances are being made regarding hardware and intelligent software to support autonomous actions of robots, less emphasis has been put on designing robot behavior that is comprehensible and pleasant for older adults. However, good usability and user experience are crucial factors for acceptance and long-term use. One way to actively engage older adults in behavioral design for social robots is participatory design. The Modality Card Deck is proposed, a tool that helps to engage older adults in human-robot interaction design process and participate in design decision for robot behavior. The cards guide the users towards creating ideas for design solutions which are detailed enough to be implemented by interaction designers and software developers. This paper provides a detailed description of the Modality Card Deck and presents an evaluation of the tool in the scope of a case study. In the case study, the card deck was used in participatory design workshops with older adults to develop multi-modal robot behaviors for the Pepper robot and a quiz game application. After describing the procedure of the case study, the workshop results and learnings about working with the Modality Card Deck and older adults are presented.

Multimodal pipe-climbing robot with origami clutches and soft modular legs

In nature, climbing trees and pipes of varying diameters or even navigating inside of hollow pipes and tree holes is easy for some climbing animals and insects. However, today’s pipe-climbing robots, which are important for automatically conducting periodic inspections and maintenance of pipelines to save time and keep humans away from hazardous environments, are designed mainly for a specific task, limiting their adaptability to different working scenarios and further implementation in real-life. In this paper, we propose a pipe-climbing robot with a soft linear actuator for bioinspired propulsion, two origami clutches to realize multi-degrees-of-freedom (DoF) motion and two pairs of soft modular legs for multimodal climbing. Design, modeling and experimental validation of the origami clutch are introduced in detail. Preliminary experimental results show that we can achieve a stroke of up to 289.6% and a maximum 45° bending angle on the soft linear actuator by regulating the air pressure inside the soft actuator and origami clutches. Additionally, by choosing the leg-type, three climbing modes, including out-pipe versatile mode, out-pipe high-force mode and in-pipe mode can be realized for particular working scenarios. A prototype climbing robot demonstrates that in out-pipe versatile mode, the robot can climb on the exterior of pipes made of various materials including PVC, rubber and metal with diameters ranging from 105 to 117 mm. In the out-pipe high-force mode, the climber can navigate along a specific pipe carrying maximum 675 g external load at the top or 200 g hanging from the bottom, as well as keeping functional without failure under static loads as high as 1968 g. In the in-pipe mode, the robot is able to travel inside pipes. This research might bridge the design gap between in-pipe and out-pipe climbing robots while offering an alternative option for soft robots to execute multi-DoF motion.

Multimodal Soft Robot for Complex Environments Using Bionic Omnidirectional Bending Actuator

Soft robots have the characteristics of good adaptability to an environment. But at present, most soft robots only complete a single specific task, and as a result, their ability to adapt to complex environments is limited. In order to solve the weak ability of soft robots to adapt to complex and diverse environments, this paper presents a new bionic omnidirectional bending actuator (BOBA) inspired by leeches, caterpillars, and other mollusks. BOBA has less radial expansion, fast response, good flexibility, and omnidirectional bending. Using modular methods, it can be quickly assembled into a soft robot with a variety of motion modes. Experiments show the multimodal soft robot can climb vertical walls, transition from one surface to another, climb stairs, leap obstacles, cross narrow pipes, and move with heavy objects. This greatly improves the ability of soft robots to adapt to complex and diverse environments.

Implementation of a Reconfigurable Robot to Achieve Multimodal Locomotion Based on Three Rules of Configuration

SUMMARYIn this paper, we focus on the configuration design of a reconfigurable robot that merges the functions of wheels, tracks, and legs together. A deformable rim is utilized to make the robot wheel reconfigurable to change its locomotion mode. Three rules of configuration design to achieve reconfiguration between different modes are proposed: (1) in wheel mode, the track wheel set should be hidden inside the wheel rim; (2) in track/leg mode, the folded wheel rim should be hidden inside the caterpillar loop; (3) the circumference of the wheel rim in wheel mode should be equal to the length of the track ring in track mode. According to these rules, the configuration of the deformable rim, track wheel set, and telescopic spoke are analyzed and designed. A prototype of the reconfigurable wheel is fabricated by three-dimensional printing, and its functions of locomotion in different modes, the switch between different modes, and its load-bearing ability are tested, verifying the effectiveness of the configuration design. Furthermore, a prototype of the reconfigurable robot is manufactured by computerized numerical control (CNC) machining to verify the structural design of the reconfigurable wheel. Compared to traditional hybrid robots with separate wheels, tracks, and legs, this reconfigurable design lends the multimodal robot both excellent terrain adaptability and a compact structure; thus, it can be widely used as a universal mobile platform in search and rescue missions and explosive object disposal missions.

SMOOTH Robot: Design for a Novel Modular Welfare Robot

Demographic change is expected to challenge many societies in the next few decades if todays’ standards of services in e.g. elder care shall be maintained. Robots are considered to at least partially mitigate this challenge, however, robots are rarely applied in the welfare domain. This paper describes the development of a concept for a novel welfare robot that is modular and affordable. The development is based on a participatory design process and by taking strengths and limitations of selected, commercially available robots into account. This work contributes a design methodology specific for welfare robots and a resulting robot concept that address three use cases in a care center. The concept includes multi-modal robot perception that facilitates a proactive robot behavior for achieving smooth interactions with end-users.

Evaluation of a Large Scale Event Driven Robot Skin

This letter evaluates and describes the large-scale integration of our multi-modal event-driven robot skin system on our humanoid robot H1 (REEM-C, PAL robotics). The robot skin is powered by the robot and all processing of tactile perception and control are executed onboard the robot. The robot skin system largely covers the humanoid and employs 1260 skin cells in 47 skin patches, in total 7560 multi-modal tactile sensors. The robot skin system is driven by events, i.e., the occurrence of novel information drives the acquisition, transmission, and processing of information rather than the synchronous sampling clock as in clock-driven systems. The event-driven robot skin system enables efficient multi-modal large area tactile perception, which is a prerequisite for realizing reactive whole-body control. We analyze the new robot skin system and evaluate its performance in clock-driven mode and event-driven mode. This analysis includes modeling the required CPU load and investigating the scaling of the system in multi-core systems. We evaluate the robot skin system with an experiment, where a large number of skin cells is activated (>680 cells), and a large number of events is generated. The obtained results demonstrate the superior performance of the event-driven system. In the clock-driven mode, the robot skin system constantly produces 315 000 packets/s, while in event-driven mode the system at most produces 40 000 packets/s (13%). The CPU load reduces from constantly 270% to at most 100% (37%). In clock-driven mode, the PC drops on average 80 000 packets/s (25% of all packets), while in event-driven mode, the package loss is practically neglectable. This efficient large-scale robot skin enables the complete onboard integration into a humanoid robot without the need for additional external power or processing capabilities.

Development and experiments of a bio-inspired robot with multi-mode in aerial and terrestrial locomotion

This paper introduces a new multi-modal robot capable of terrestrial and aerial locomotion, aiming to operate in a wider range of environments. The robot was built to achieve two locomotion modes of walking and gliding while preventing one modality hindering the other. To achieve this goal, we found the solution from Pteromyini, commonly known as the flying squirrel. Pteromyini utilizes its flexible membrane to glide in the air, and it shows agile movements on the ground. We studied Pteromyini to mimic the key features that allow Pteromyini to perform aerial and terrestrial locomotion. We adopted the flexible membrane and gliding strategy of Pteromyini to the robot. Through dynamics analysis and simulations, the overall design was determined. The flexibility of the membrane was also chosen considering the robot’s performance in the air and on the ground. The leg was optimized to perform with regulated motor torques in both walking and gliding. From gliding tests, the robot showed an average gliding ratio of 1.88. Inspired by Pteromyini, controlling the robot’s angle of attack with leg and tail movement was also adopted and tested. Different gait patterns and changing walking directions were tested to demonstrate the robot’s terrestrial performance. The average walking speed was 13.38 cm s−1. The experimental results demonstrated the robot’s functionality in aerial and terrestrial locomotion.

Specifying Dual-Arm Robot Planning Problems Through Natural Language and Demonstration

Multi-modal robot programming with natural language and demonstration is a promising technique for efficient teaching of manipulation tasks in industrial environments. In particular, with modern dual-arm robots designed to quickly take over tasks at typical industrial workbenches, the direct teaching of task sequences hardly utilizes the robots’ capabilities. We therefore propose a two-staged approach that combines natural language instructions and demonstration with simultaneous task allocation and motion scheduling based on constraint programming. Instead of providing a task description and demonstrations that are replayed to a large extent, the user describes tasks to be scheduled with all relevant constraints and demonstrates relevant locations relative to workpieces and other objects. With explicitly stated constraints on the partial ordering of tasks, the solver allocates the tasks to the robot arms and schedules them in time while avoiding self-collisions and reducing the makespan in our experiment by $33\%$ . The linguistic concepts of naming and grouping enable systematic reuse of sub-task ensembles. The proposed approach is evaluated with four variants of a gluing use-case from furniture assembly in user studies with ten participants. In these user studies, we observed a speed-up for the task definition of more than six times compared to a textual specification of the planning problems using the Python-based planner API.

JumpRoACH: A Trajectory-Adjustable Integrated Jumping–Crawling Robot

In this paper, we present a milliscale-integrated jumping–crawling robot that can adjust its launch trajectory and upright itself. This multimodal robot shows an enhanced performance of overcoming obstacles compared with a robot with a single locomotion mode. To make this possible, the robot uses a newly developed jumping module with improved energy storing capacity and a height-adjustable active clutch. The jumping module utilizes both linear springs and torsional springs to maximize the energy-storing capacity under the given limit of structural robustness. To adjust the quantity of stored energy and release the energy at any state, an active clutch mechanism based on a single dc motor is developed, which enables the robot to control both jump timing and height. Also, an active shell allows the robot to upright itself after landing and to continue jumping and crawling. The jumping module and the shell are integrated with the lightweight VelociRoACH crawler. To show the usability in real-world applications, the integrated jumping–crawling robot is tested on the cluttered terrain. In result, the robot reaches a target using jumping, crawling, and self-righting.

Passive force control of multimodal astronaut training robot

For the purpose of solving the problem of astronaut training in weightlessness environment, this article proposes a multimodal astronaut training robot to enable astronauts to perform running, bench press and deep squat training in the weightless environment, so as to help them mitigate the adverse effects brought by the space adaptation syndrome. Taking the modularized wire driving unit as the research object, the dynamic model of the passive force servo system was established; and the passive force control strategy was designed. The experimental results show that the system is of good stability, high steady-state accuracy, and excellent dynamic quality after correction. When the given signal frequency is 10 Hz, the system phase lag is about 9°, and the loading error is about 5%. The passive force servo control strategy can effectively reduce the surplus force. When the speed disturbance frequency of carrying unit is within 3 Hz, the elimination rate of the surplus force can reach 90%.

A data-efficient deep learning approach for deployable multimodal social robots

The deep supervised and reinforcement learning paradigms (among others) have the potential to endow interactive multimodal social robots with the ability of acquiring skills autonomously. But it is still not very clear yet how they can be best deployed in real world applications. As a step in this direction, we propose a deep learning-based approach for efficiently training a humanoid robot to play multimodal games—and use the game of ‘Noughts and Crosses’ with two variants as a case study. Its minimum requirements for learning to perceive and interact are based on a few hundred example images, a few example multimodal dialogues and physical demonstrations of robot manipulation, and automatic simulations. In addition, we propose novel algorithms for robust visual game tracking and for competitive policy learning with high winning rates, which substantially outperform DQN-based baselines. While an automatic evaluation shows evidence that the proposed approach can be easily extended to new games with competitive robot behaviours, a human evaluation with 130 humans playing with the Pepper robot confirms that highly accurate visual perception is required for successful game play.

Design and Simulation Optimization of Obstacle Avoidance System for Planetary Exploration Mobile Robots

This paper proposes a new type of planetary detection multi-modal motion robot based on pure mechanical structure. The detection robot does not contain traditional sensors, computers and power supply systems, and uses pure mechanical components such as gears, friction wheels, rack and pinion mechanisms, and cam mechanisms to perform functions such as energy generation, transmission, and obstacle avoidance. In this paper, the obstacle avoidance system of the mobile robot is designed and optimized for the core components.

Central Pattern Generator (CPG) Control of a Biomimetic Robot Fish for Multimodal Swimming

This paper introduces the design and control of a biomimetic robot fish for multimodal swimming. The biomimetic design consists of three parts: the rigid head, the wire-driven body and the compliant tail. The control is an improved Central Pattern Generator (CPG) with the high-level control command: (M, ω, B, R), where M is the amplitude, ω is the angular velocity, B is the offset and R is the time ratio between two phases forming one flapping cycle. This method differs from previous research in two aspects: (1) The CPG control is firstly implemented on the wire-driven robot fish. (2) The improved CPG model synthesizes symmetrical flapping in cruising and asymmetrical flapping in turning for the robot fish. The asymmetrical flapping refers to the asymmetry of the offset and the time ratio. This combination of the design and the control has several advantages over the existing multimodal swimming robot fishes. First, it uses just one driving motor for undulatory oscillation while the others need to use two or more motors. Second, with just one motor, the CPG control can be easily implemented. Third, the use of the time ratio, R, makes the robot fish turn more naturally and effectively. Experimental results show the robot fish achieved the maximum speed of 1.37 Body Length/Second (BL·s-1) and the largest turning rate of 457°/s. Additionally, in many swimming conditions, its Strouhal Number falls in the range from 0.2 to 0.4, which implies the robot fish is efficient.

Aerodynamic Characteristics and Pitching Adjusting Mechanism of the Flying Squirrel With Deployed Patagium

The flying squirrel has a balanced amphibious ability on land and air, especially the ability to glide using their patagium, which makes it an excellent biomimetic object in the design and development of multimodal locomotion robots. This paper presents the study of aerodynamic characteristics and pitching adjusting mechanism of the flying squirrel with deployed patagium. A 3D geometric model of the flying squirrel with unfolded wing membrane was obtained using 3D scanning technique. Then a simulation analysis based on computational fluid dynamics software and a wind tunnel experiment on the animal specimen were carried out to analyze and evaluate the aerodynamic characteristics. After that, the pitching adjusting mechanism of flying squirrels during gliding was numerical simulated by establishing the pitching dynamics model. The results from above research not only enrich our understanding of gliding mechanism of flying squirrels, but also provide important biological inspiration for the design and development of multimodal robots in the future.