Robot Motion Generation Research Articles

Unleashing mixed-reality capability in Deep Reinforcement Learning-based robot motion generation towards safe human–robot collaboration

The integration of human–robot collaboration yields substantial benefits, particularly in terms of enhancing flexibility and efficiency within a range of mass-personalized manufacturing tasks, for example, small-batch customized product inspection and assembly/disassembly. Meanwhile, as human–robot collaboration lands broader in manufacturing, the unstructured scene and operator uncertainties are increasingly involved and considered. Consequently, it becomes imperative for robots to execute in a safe and adaptive manner rather than solely relying on pre-programmed instructions. To tackle it, a systematic solution for safe robot motion generation in human–robot collaborative activities is proposed, leveraging mixed-reality technologies and Deep Reinforcement Learning. This solution covers the entire process of collaboration starting with an intuitive interface that facilitates bare-hand task command transmission and scene coordinate transformation before the collaboration begins. In particular, mixed-reality devices are employed as effective tools for representing the state of humans, robots, and scenes. This enables the learning of an end-to-end Deep Reinforcement Learning policy that addresses both the uncertainties in robot perception and decision-making in an integrated manner. The proposed solution also implements policy simulation-to-reality deployment, along with motion preview and collision detection mechanisms, to ensure safe robot motion execution. It is hoped that this work could inspire further research in human–robot collaboration to unleash and exploit the powerful capabilities of mixed reality.

Enhancing Robot Task Planning and Execution through Multi-Layer Large Language Models.

Large language models have found utility in the domain of robot task planning and task decomposition. Nevertheless, the direct application of these models for instructing robots in task execution is not without its challenges. Limitations arise in handling more intricate tasks, encountering difficulties in effective interaction with the environment, and facing constraints in the practical executability of machine control instructions directly generated by such models. In response to these challenges, this research advocates for the implementation of a multi-layer large language model to augment a robot's proficiency in handling complex tasks. The proposed model facilitates a meticulous layer-by-layer decomposition of tasks through the integration of multiple large language models, with the overarching goal of enhancing the accuracy of task planning. Within the task decomposition process, a visual language model is introduced as a sensor for environment perception. The outcomes of this perception process are subsequently assimilated into the large language model, thereby amalgamating the task objectives with environmental information. This integration, in turn, results in the generation of robot motion planning tailored to the specific characteristics of the current environment. Furthermore, to enhance the executability of task planning outputs from the large language model, a semantic alignment method is introduced. This method aligns task planning descriptions with the functional requirements of robot motion, thereby refining the overall compatibility and coherence of the generated instructions. To validate the efficacy of the proposed approach, an experimental platform is established utilizing an intelligent unmanned vehicle. This platform serves as a means to empirically verify the proficiency of the multi-layer large language model in addressing the intricate challenges associated with both robot task planning and execution.

Wiring connector-terminated cables based on manipulation planning with collision-free EMD net

In this paper, we propose a manipulation planning method for cable wiring in the assembly of electric appliances etc. We address a scenario where a robot grasps a connector attached to the end of a cable and has to bring the connector to a socket. To accomplish this automatically, we propose a novel manipulation planning method. The method extends the Encode-Manipulate-Decode network (EMD net), which can predict shape changes of deformable objects and generate robot motion sequences for producing desired shape transitions. This enables us to find connector trajectories that avoid collision between the cable and the surrounding environment. We conducted experiments with several different cable lengths. We also introduce some functions required for real-world wiring, such as online cable shape modification. Experimental results show that the proposed method can achieve stable manipulation of real cables.

A novel teacher–student hierarchical approach for learning primitive information

In this study, we focus on the challenge of robot motion generation, which is critical for improving the ability of robots to perform downstream tasks. To this end, inspired by the idea of variational inference, we propose a novel teacher–student hierarchical approach for learning continuous primitive spaces to generate robot motions. We first learn a variational auto-encoder (VAE) based on Gated Recurrent Unit (GRU) from demonstrations, the encoder is used as a teacher model to encode trajectories to primitive information, and the decoder aims to decode latent information to joint trajectories. To learn a primitive action space for the high-level model, we propose a novel learning method to train a student model. We utilize the trained teacher to encode trajectories to labels, and use the Kullback–Leibler (KL) divergence and the reconstruction errors between the label and the predicted primitive information as the loss to train a student model. We then utilize trained student models to guide high-level policy learning. We evaluate our approach on three different robot datasets. Experiments demonstrate that our method obtains the ability to generate new motion and discover common information across tasks, it also can be used to accelerate downstream task learning.

Modality Attention for Prediction-Based Robot Motion Generation: Improving Interpretability and Robustness of Using Multi-Modality

Modality Attention for Prediction-Based Robot Motion Generation: Improving Interpretability and Robustness of Using Multi-Modality

Sampling-Based Motion Planning: A Comparative Review

Sampling-based motion planning is one of the fundamental paradigms to generate robot motions, and a cornerstone of robotics research. This comparative review provides an up-to-date guide and reference manual for the use of sampling-based motion planning algorithms. It includes a history of motion planning, an overview of the most successful planners, and a discussion of their properties. It also shows how planners can handle special cases and how extensions of motion planning can be accommodated. To put sampling-based motion planning into a larger context, a discussion of alternative motion generation frameworks highlights their respective differences from sampling-based motion planning. Finally, a set of sampling-based motion planners are compared on 24 challenging planning problems in order to provide insights into which planners perform well in which situations and where future research would be required. This comparative review thereby provides not only a useful reference manual for researchers in the field but also a guide for practitioners to make informed algorithmic decisions.

A Method of Generating Robot Motion using Common Motion Trajectory Model and Function Information of Tools

A Method of Generating Robot Motion using Common Motion Trajectory Model and Function Information of Tools

A fast optical proximity sensor skin that contains an analog computing circuit and can cover an entire link

This paper proposes a novel optical proximity sensor skin that can cover entire robot links and has high responsiveness, wire-saving properties, and suitability for covering narrow surfaces. Moreover, it can extract information that can help to control robots. Although implementing these characteristics simultaneously has been problematic for sensor skins covering links, the proposed sensor skin achieves this using a newly developed analog computing circuit. This analog computing circuit only consists of several resistors and diodes and is based on the splitting ratio of photocurrents variation based on the design of the resistance values. If the output from the proximity sensing element is interpreted as a virtual force, the processed output denotes its magnitude and the location of the point of action. Even if multiple objects are detected, the outputs are associated with the resultant virtual force because photocurrents are merged. Therefore, the characteristics such as high responsiveness are maintained. This work fabricated a prototype to evaluate the proposed sensor skin. Then, collision avoidance experiments against human hands were conducted to validate the efficiency of the proposed sensor skin for generating robot motions. These results verified that the proposed sensor skin is suitable for robots reacting quickly to unknown environments.

Automatic welding sequence and robot motion generation technology for automated teaching of welding robots

Automatic welding sequence and robot motion generation technology for automated teaching of welding robots

Touch Semantics for Intuitive Physical Manipulation of Humanoids

Rather than systematically programming joint or task trajectories, having a human physically manipulate the robot for direct adjustments is more intuitive, saves time, and increases usability, especially for nonexperts. Interactive motion generation or repositioning of humanoid robots through direct human-touch manipulation is not an easy task, especially for high-level multijoint maneuvers. We propose a set of design rules for generating intuitive touch semantics called the “two-touch kinematic chain paradigm.” Our method interprets user touch intentions to allow motions ranging from low-level single joint control to high-level whole-body task control with posture generation, stepping, and walking. The goal is to provide the user with an intuitive protocol for physical humanoid manipulation that can serve the purpose of any application. The generated set of touch semantics is embodied in a finite state machine-based framework using a task-space quadratic programming controller to interpret human touch using capacitive sensors embedded in the humanoid shell, and force-torque sensors located at the ankles and wrists. A position-controlled humanoid robot is used to assess the utility and function of our proposed touch semantics for physical manipulation. Furthermore, a user study with nonexperts examines how our approach is perceived in practice.

Graph-Based Framework on Bimanual Manipulation Planning from Cooking Recipe

It is difficult to effectively operate a dual-arm robot using only the information written in a cooking recipe. To cope with this problem, this paper proposes a graph-based approach on bimanual cooking motion planning from a cooking recipe. In our approach, we first decompose the cooking recipe into graph elements. Then, we try to connect the graph elements taking into account the attributes of the input/output nodes. If two graph elements cannot be connected to each other, we search for a graph element that can be inserted between them from a motion database. Since the constructed graph includes the whole sequence of the robot’s motions to perform the cooking task, we can generate a task sequence of a dual-arm manipulator simultaneously performing two different tasks by using two arms. Through experimental study, we show that it is possible to generate robot motions from a cooking recipe and perform the cooking motions while simultaneously moving the left and right arms.

Unified Solution of Different-Kind Future Matrix Equations Using New Nine-Instant Discretization Formula and Zeroing Neural Dynamics

In this article, time-varying matrix equation problems, including the Lyapunov equation, matrix inversion, and generalized matrix inversion are investigated in a future (or say, discrete time-varying) perspective. Then, in order to develop a unified solution model for the above three future problems, a future matrix equation (FME) is investigated. The discrete-time unified solution (DTUS) model, which is based on the zeroing neural dynamics (ZND) method and a new nine-instant Zhang <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> discretization (ZeaD) formula, is thus proposed and termed the nine-instant DTUS (9IDTUS) model. Meanwhile, theoretical analyses on the stability and precision of the 9IDTUS model are provided. In addition, conventional DTUS models obtained from the Euler forward formula, Taylor–Zhang discretization formula, and a seven-instant discretization formula are also presented for comparisons. Furthermore, numerical experiments including the robot motion generation, are conducted and analyzed to substantiate the efficacy and superiority of the proposed 9IDTUS model.

Deep Active Visual Attention for Real-Time Robot Motion Generation: Emergence of Tool-Body Assimilation and Adaptive Tool-Use

Sufficiently perceiving the environment is a critical factor in robot motion generation. Although the introduction of deep visual processing models have contributed in extending this ability, existing methods lack in the ability to actively modify what to perceive; humans perform internally during visual cognitive processes. This letter addresses the issue by proposing a novel robot motion generation model, inspired by a human cognitive structure. The model incorporates a state-driven active top-down visual attention module, which acquires attentions that can actively change targets based on task states. We term such attentions as role-based attentions, since the acquired attention directed to targets that shared a coherent role throughout the motion. The model was trained on a robot tool-use task, in which the role-based attentions perceived the robot grippers and tool as identical end-effectors, during object picking and object dragging motions respectively. This is analogous to a biological phenomenon called tool-body assimilation, in which one regards a handled tool as an extension of one’s body. The results suggested an improvement of flexibility in model’s visual perception, which sustained stable attention and motion even if it was provided with untrained tools or exposed to experimenter’s distractions.

Social Momentum: Design and Evaluation of a Framework for Socially Competent Robot Navigation

Mobile robots struggle to integrate seamlessly in crowded environments such as pedestrian scenes, often disrupting human activity. One obstacle preventing their smooth integration is our limited understanding of how humans may perceive and react to robot motion. Motivated by recent studies highlighting the benefits of intent-expressive motion for robots operating close to humans, we describe Social Momentum (SM), a planning framework for legible robot motion generation in multiagent domains. We investigate the properties of motion generated by SM via two large-scale user studies: an online, video-based study ( N = 180) focusing on the legibility of motion produced by SM and a lab study ( N = 105) focusing on the perceptions of users navigating next to a robot running SM in a crowded space. Through statistical and thematic analyses of collected data, we present evidence suggesting that (a) motion generated by SM enables quick inference of the robot’s navigation strategy; (b) humans navigating close to a robot running SM follow comfortable, low-acceleration paths; and (c) robot motion generated by SM is positively perceived and indistinguishable from a teleoperated baseline. Through the discussion of experimental insights and lessons learned, this article aspires to inform future algorithmic and experimental design for social robot navigation.

Fault-tolerant motion planning and generation of quadruped robots synthesised by posture optimization and whole body control

Quadruped robots are likely to fall into the fault joint state in outdoor explorations. The unexpected joint lock may suddenly happen when quadruped robots are implementing normal gaits, and maintaining the primal movement patterns to finish targeted tasks could be disastrous. In this paper, a fault-tolerant motion planning and generation method for quadruped robots with joint lock is proposed. Fault-tolerant cases on three types of joints on legs are investigated, and equivalent geometric models are proposed to reconstruct kinematics. To make unnecessary deformation of gait patterns as small as possible, the body posture and standing height of quadruped robots are to be optimized based on the nonlinear equivalent geometry models with constraints. The proposed fault-tolerant method is applicable to constructing a quasi-static whole-body controller, and it does not require additional operations and constraints of the fault leg. To validate the consistency and stability of the proposed fault-tolerant method, the experiments are implemented on the three joint lock failure scenarios for quadruped robots.

A COMPUTATIONAL FRAMEWORK OF GOAL DIRECTED VOLUNTARY MOTION GENERATION AND CONTROL LOOP IN HUMANOID ROBOTS

In this paper, it is aimed to construct a computational framework related to bio-inspired motion generation and control systems for humanoid robots. To acquire natural motion patterns in humanoid robots, behaviors observed from biological motor systems in humans and other mammals should be analyzed in detail. Computational mechanisms are mainly placed on the bio-physical plausible neural structures embodied in different dynamics. The main components of the system are composed of the limbic system, neocortex, cerebellum, brainstem, and spinal cord modules. Internal dynamics of these modules include a nonlinear estimator (e.g. chaotic attractor), memory formation, learning (neural plasticity) procedure. While the proposed novel neuro-cognitive framework is performing goal-directed voluntary motion generation and control tasks, also it estimates the amount of motion errors and computes motion correction signals. By this study, some motion-based central nervous system lesions (e.g. epilepsy, Parkinson, etc.) can be computationally modeled so that impairments of motor control commands are detected. Thus motion disorders can be reconstructed not only in humanoid robots but also in humans via some locomotion equipment.

A deep learning framework for realistic robot motion generation

A deep learning framework for realistic robot motion generation

Packing Planning and Execution Considering Arrangement Rules

To send a consumer the ordered products in e-commerce, it is necessary to pack the products into a container. When packing, we need to consider the arrangement of the products to avoid damaging them during transportation by using the know-how of packing. In this paper, we propose a packing robot system that plans the arrangement and then executes it. To develop this robot system, we employ two ideas. The first idea is to split the whole process into the arrangement of the objects and robot motion generation. The arrangement is regarded as a 3D bin-packing problem. The second idea is to use the heuristics of the packing so that objects are stacked from the bottom. Since the space above the target position is free from collisions, the strategy to approach from the top reduces the possibility of a collision. In the experiments, we succeeded in arranging eleven objects satisfying the assigned rules and packing ten of the eleven objects using the robot.

RMPflow: A Geometric Framework for Generation of Multitask Motion Policies

Generating robot motion for multiple tasks in dynamic environments is challenging, requiring an algorithm to respond reactively while accounting for complex nonlinear relationships between tasks. In this article, we develop a novel policy synthesis algorithm, Riemannian motion policy (RMP)flow, based on geometrically consistent transformations of RMPs. RMPs are a class of reactive motion policies that parameterize non-Euclidean behaviors as dynamical systems in intrinsically nonlinear task spaces. Given a set of RMPs designed for individual tasks, RMPflow can combine these policies to generate an expressive global policy, while simultaneously exploiting sparse structure for computational efficiency. We study the geometric properties of RMPflow and provide sufficient conditions for stability. Finally, we experimentally demonstrate that accounting for the natural Riemannian geometry of task policies can simplify classically difficult problems, such as planning through the clutter on high-degree-of-freedom manipulation systems. Note to Practitioners —Requirements on safety and responsiveness for collaborative robots have driven a need for new ideas in control design that bridge between standard objectives in low-level control (such as trajectory tracking) and high-level behavioral objectives (such as collision avoidance) often relegated to planning systems. Modern results from geometric control, which promise stable controllers that can smoothly and safely transition between many behavioral tasks, therefore, become highly relevant. However, for years, this field has remained inaccessible due to its mathematical complexity. This article aims to: 1) make those ideas accessible to robotics and control experts by recasting them in a concrete algorithmic framework amenable to controller design and 2) to additionally generalize them to better satisfy the specific needs of robotic behavior generation. Our experiments demonstrate that the resulting controllers can engender natural behavior that adapts instantaneously to changing surroundings with zero planning while performing manipulation tasks. The framework is gaining traction within the robotics community, finding increasing application in areas, such as autonomous navigation, tactile servoing, and multi-agent systems. Future research will address learning these controllers from data to simplify that process of design and tuning, which at present can require experience.

Robot Motion Generation by Hand Demonstration

Robot Motion Generation by Hand Demonstration