Robot Learning Research Articles

A systematic review on robot-assisted language learning for adults

In the 21st-century era of globalization, language proficiency is a pivotal connector across cultures, with artificial intelligence (AI) revolutionizing educational paradigms through Robot-Assisted Language Learning (RALL). This systematic review examines the role of RALL in adult second language acquisition, focusing on its pedagogical strategies and learner engagement. Unlike the previous systematic reviews that explore the multifaceted roles of robots in language learning, including as teachers, tutors, assistants, and peer learners, we identify explicit and implicit instructional strategies within RALL, highlighting the unique learning landscape of adult learners characterized by self-regulation and self-direction. We assess the latest advancements in RALL for adult learners through three research questions, compare the effectiveness of explicit versus implicit instructions, and investigate affective factors enhancing RALL performance. Our review contributes a comprehensive status analysis, in-depth exploration of interaction modes, and insights for future research directions, providing a roadmap for academic research and practical guidance for educators and robot developers. This study aims to optimize RALL strategies to better meet the needs of adult learners, fostering a more efficient and engaging language learning experience.

Robot learning on the job: Human-in-the-loop autonomy and learning during deployment

With the rapid growth of computing powers and recent advances in deep learning, we have witnessed impressive demonstrations of novel robot capabilities in research settings. Nonetheless, these learning systems exhibit brittle generalization and require excessive training data for practical tasks. To harness the capabilities of state-of-the-art robot learning models while embracing their imperfections, we present Sirius, a principled framework for humans and robots to collaborate through a division of work. In this framework, partially autonomous robots are tasked with handling a major portion of decision-making where they work reliably; meanwhile, human operators monitor the process and intervene in challenging situations. Such a human–robot team ensures safe deployments in complex tasks. Further, we introduce a new learning algorithm to improve the policy’s performance on the data collected from the task executions. The core idea is re-weighing training samples with approximated human trust and optimizing the policies with weighted behavioral cloning. We evaluate Sirius in simulation and on real hardware, showing that Sirius consistently outperforms baselines over a collection of contact-rich manipulation tasks, achieving an 8% boost in simulation and 27% on real hardware than the state-of-the-art methods in policy success rate, with twice faster convergence and 85% memory size reduction. Videos and more details are available at https://ut-austin-rpl.github.io/sirius/ .

A Human-Centered View of Continual Learning: Understanding Interactions, Teaching Patterns, and Perceptions of Human Users Toward a Continual Learning Robot in Repeated Interactions

Continual learning (CL) has emerged as an important avenue of research in recent years, at the intersection of Machine Learning (ML) and Human–Robot Interaction (HRI), to allow robots to continually learn in their environments over long-term interactions with humans. Most research in CL, however, has been robot-centered to develop CL algorithms that can quickly learn new information on systematically collected static datasets. In this article, we take a human-centered approach to CL, to understand how humans interact with, teach, and perceive CL robots over the long term, and if there are variations in their teaching styles. We developed a socially guided CL system that integrates CL models for object recognition with a mobile manipulator robot and allows humans to directly teach and test the robot in real time over multiple sessions. We conducted an in-person study with 60 participants who interacted with the CL robot in 300 sessions with 5 sessions per participant. In this between-participant study, we used three different CL models deployed on a mobile manipulator robot. An extensive qualitative and quantitative analysis of the data collected in the study shows that there is significant variation among the teaching styles of individual users indicating the need for personalized adaptation to their distinct teaching styles. Our analysis shows that the constrained experimental setups that have been widely used to test most CL models are not adequate, as real users interact with and teach CL robots in a variety of ways. Finally, our analysis shows that although users have concerns about CL robots being deployed in our daily lives, they mention that with further improvements CL robots could assist older adults and people with disabilities in their homes.

Planning the Movement of Robots in a Social Environment Via Reinforcement Learning

Planning the Movement of Robots in a Social Environment Via Reinforcement Learning

Diffusion policy: Visuomotor policy learning via action diffusion

This paper introduces Diffusion Policy, a new way of generating robot behavior by representing a robot’s visuomotor policy as a conditional denoising diffusion process. We benchmark Diffusion Policy across 15 different tasks from 4 different robot manipulation benchmarks and find that it consistently outperforms existing state-of-the-art robot learning methods with an average improvement of 46.9%. Diffusion Policy learns the gradient of the action-distribution score function and iteratively optimizes with respect to this gradient field during inference via a series of stochastic Langevin dynamics steps. We find that the diffusion formulation yields powerful advantages when used for robot policies, including gracefully handling multimodal action distributions, being suitable for high-dimensional action spaces, and exhibiting impressive training stability. To fully unlock the potential of diffusion models for visuomotor policy learning on physical robots, this paper presents a set of key technical contributions including the incorporation of receding horizon control, visual conditioning, and the time-series diffusion transformer. We hope this work will help motivate a new generation of policy learning techniques that are able to leverage the powerful generative modeling capabilities of diffusion models. Code, data, and training details are available (diffusion-policy.cs.columbia.edu).

Surgical Experience and Functional Outcomes after Laparoscopic and Robot-Assisted Partial Nephrectomy: Results from a Multi-Institutional Collaboration

Background: In patients treated with partial nephrectomy, prior evidence showed that peri-operative outcomes, such as complications and ischemia time, improved as a function of the surgical experience of the surgeon, but data on functional outcomes after surgery are still scarce. Methods: We retrospectively analyzed data of 4011 patients with a single, unilateral cT1a-b renal mass treated with laparoscopic or robot-assisted partial nephrectomy. The operations were performed by 119 surgeons at 22 participating institutions between 1997 and 2022. Multivariable models investigated the association between surgical experience (number of prior operations) and acute kidney injury (AKI) and recovery of at least 90% of baseline estimated glomerular filtration rate (eGFR) 1 yr after partial nephrectomy. The adjustment for case mix included age, Body Mass Index, preoperative serum creatinine, clinical T stage, PADUA score, warm ischemia time, pathologic tumor size, and year of surgery. Results: A total of 753 (19%) and 3258 (81%) patients underwent laparoscopic and robot-assisted partial nephrectomy, respectively. Overall, 37 (31%) and 55 (46%) surgeons contributed only to laparoscopic and robotic learning curves, respectively, whereas 27 (23%) contributed to the learning curves of both approaches. In the laparoscopic group, 8% and 55% of patients developed AKI and recovered at least 90% of their baseline eGFR, respectively. After adjusting for confounders, we did not find evidence of an association between surgical experience and AKI after laparoscopic partial nephrectomy (odds ratio [OR]: 0.9992; 95% confidence interval [CI]: 0.9963, 1.0022; p = 0.6). Similar results were found when 1-year renal function was the outcome of interest (OR: 0.9996; 95% CI: 0.9988, 1.0005; p = 0.5). Among patients who underwent robot-assisted partial nephrectomy, AKI occurred in 11% of patients, whereas 54% recovered at least 90% of their baseline eGFR. On multivariable analyses, the relationship between surgical experience and AKI after surgery was not statistically significant (OR: 1.0015; 95% CI: 0.9992, 1.0037; p = 0.2), with similar results when the outcome of interest was renal function one year after surgery (OR: 1.0001; 95% CI: 0.9980, 1.0022; p = 0.9). Virtually the same findings were found on sensitivity analyses. Conclusions: In patients treated with laparoscopic or robot-assisted partial nephrectomy, our data suggest that the surgical experience of the operating surgeon might not be a key determinant of functional recovery after surgery. This raises questions about the use of serum markers to assess functional recovery in patients with two kidneys and opens the discussion on what are the key steps of the procedure that allowed surgeons to achieve optimal outcomes since their initial cases.

A survey of communicating robot learning during human-robot interaction

For robots to seamlessly interact with humans, we first need to make sure that humans and robots understand one another. Diverse algorithms have been developed to enable robots to learn from humans (i.e., transferring information from humans to robots). In parallel, visual, haptic, and auditory communication interfaces have been designed to convey the robot’s internal state to the human (i.e., transferring information from robots to humans). Prior research often separates these two directions of information transfer, and focuses primarily on either learning algorithms or communication interfaces. By contrast, in this survey we take an interdisciplinary approach to identify common themes and emerging trends that close the loop between learning and communication. Specifically, we survey state-of-the-art methods and outcomes for communicating a robot’s learning back to the human teacher during human-robot interaction. This discussion connects human-in-the-loop learning methods and explainable robot learning with multimodal feedback systems and measures of human-robot interaction. We find that—when learning and communication are developed together—the resulting closed-loop system can lead to improved human teaching, increased human trust, and human-robot co-adaptation. The paper includes a perspective on several of the interdisciplinary research themes and open questions that could advance how future robots communicate their learning to everyday operators. Finally, we implement a selection of the reviewed methods in a case study where participants kinesthetically teach a robot arm. This case study documents and tests an integrated approach for learning in ways that can be communicated, conveying this learning across multimodal interfaces, and measuring the resulting changes in human and robot behavior.

Hierarchical robot learning method for industrial fluorescent penetrant inspection

Fluorescent Penetrant Inspection (FPI) is a Non-Destructive Testing (NDT) method, extensively used to evaluate components for identifying defects across a broad range of industries. FPI process remains a manual visual inspection, where the operator by means of fluorescent dye, that penetrates discontinuities on the component, aims to distinguish between indications that are relevant (i.e., can be associated with surface defects) and non-relevant (i.e., can be associated to insufficient wash-off, dust or other non-relevant factors). The FPI process can be decomposed into the following steps: (a) thorough visual examination of the component, (b) executing manual wiping-off of the fluorescent dye with a brush, of all areas that require interrogation for potential indications, and (c) disposition of the inspected components. The number of those areas on the component that require interrogation and hence to be wiped-off is unknown a priori of the inspection and varies depending on the condition of the part. As a result, replacing this manual wipe-off step by a robot requires tedious manual programming of an excessive number of robot paths to assure reach of the robot to the entire surface of the part as well as safe robot motion. In addition, these robot motions are part specific and thus not transferrable to other geometries of components, making scaling of this technology across manufacturing industry not possible. In this paper, we propose a hierarchical robot learning method to address the challenge of reducing manual robot programming and enable the scaling of this automated NDT technology. The proposed method integrates and fuses Deep Reinforcement Learning (DRL), Screw Linear Interpolation (ScLERP) and Learning from Demonstration (LfD), enabling an autonomous generation of brushing strokes with a six-degrees of freedom (DoF) industrial manipulator, and automating the wipe-off step of the FPI process. Using this approach, a robot learning policy is generated for the wipe-off motion in a simulated industrial robotic cell at first and then the policy is transferred to the real system for validation.

Bridging vision and touch: advancing robotic interaction prediction with self-supervised multimodal learning.

Predicting the consequences of the agent's actions on its environment is a pivotal challenge in robotic learning, which plays a key role in developing higher cognitive skills for intelligent robots. While current methods have predominantly relied on vision and motion data to generate the predicted videos, more comprehensive sensory perception is required for complex physical interactions such as contact-rich manipulation or highly dynamic tasks. In this work, we investigate the interdependence between vision and tactile sensation in the scenario of dynamic robotic interaction. A multi-modal fusion mechanism is introduced to the action-conditioned video prediction model to forecast future scenes, which enriches the single-modality prototype with a compressed latent representation of multiple sensory inputs. Additionally, to accomplish the interactive setting, we built a robotic interaction system that is equipped with both web cameras and vision-based tactile sensors to collect the dataset of vision-tactile sequences and the corresponding robot action data. Finally, through a series of qualitative and quantitative comparative study of different prediction architecture and tasks, we present insightful analysis of the cross-modality influence between vision, tactile and action, revealing the asymmetrical impact that exists between the sensations when contributing to interpreting the environment information. This opens possibilities for more adaptive and efficient robotic control in complex environments, with implications for dexterous manipulation and human-robot interaction.

Active Data-enabled Robot Learning of Elastic Workpiece Interactions

Abstract During manufacturing processes, such as clamping and drilling of elastic structures, it is essential to maintain tool-workpiece normality to minimize shear forces and torques, and thereby preventing damage to the tool or the workpiece. The challenge arises in making precise model-based predictions of the relatively large deformations that occur as the applied normal force (e.g., clamping force) is increased. However, precision deformation predictions are essential for selecting the optimal robot pose that maintains force normality. Therefore, recent works have employed force-displacement measurements at each work location to determine the robot pose for maintaining tool normality. Nevertheless, this approach, which relies on local measurements at each work location and at each gradual increment of the applied normal force, can be slow and consequently, time prohibitive. The main contributions of this work are to use: Gaussian Process methods to learn the robot-pose map for force normality at unmeasured workpiece locations; active learning to optimally select and minimize the number of measurement locations needed for accurate learning of the robot-pose map. Experimental results show that the number of data points needed with active learning is 77.8% less than the case with a benchmark linear positioning learning for the same level of model precision. Additionally, the learned robot-pose map enables a rapid increase of the normal force at unmeasured locations on the workpiece, reaching force-increment rates up to eight times faster than the original force-increment rate when the robot is learning the correct pose.

Design and analysis of parallel quantum transfer fractal priority replay with dynamic memory algorithm in quantum reinforcement learning for robotics

Design and analysis of parallel quantum transfer fractal priority replay with dynamic memory algorithm in quantum reinforcement learning for robotics

FMB: A functional manipulation benchmark for generalizable robotic learning

In this paper, we propose a real-world benchmark for studying robotic learning in the context of functional manipulation: a robot needs to accomplish complex long-horizon behaviors by composing individual manipulation skills in functionally relevant ways. The core design principles of our Functional Manipulation Benchmark (FMB) emphasize a harmonious balance between complexity and accessibility. Tasks are deliberately scoped to be narrow, ensuring that models and datasets of manageable scale can be utilized effectively to track progress. Simultaneously, they are diverse enough to pose a significant generalization challenge. Furthermore, the benchmark is designed to be easily replicable, encompassing all essential hardware and software components. To achieve this goal, FMB consists of a variety of 3D-printed objects designed for easy and accurate replication by other researchers. The objects are procedurally generated, providing a principled framework to study generalization in a controlled fashion. We focus on fundamental manipulation skills, including grasping, repositioning, and a range of assembly behaviors. The FMB can be used to evaluate methods for acquiring individual skills, as well as methods for effectively combining and ordering such skills in order to solve complex, multi-stage manipulation tasks. We also offer an imitation learning framework that includes a suite of policies trained to solve the proposed tasks. This enables researchers to utilize our tasks as a versatile toolkit for examining various parts of the pipeline. For example, researchers could propose a better design for a grasping controller and evaluate it in combination with our baseline reorientation and assembly policies as part of a pipeline for solving multi-stage tasks. Our dataset, object CAD files, code, and evaluation videos can be found on our project website: https://functional-manipulation-benchmark.github.io .

Transfer learning in robotics: An upcoming breakthrough? A review of promises and challenges

Transfer learning is a conceptually-enticing paradigm in pursuit of truly intelligent embodied agents. The core concept—reusing prior knowledge to learn in and from novel situations—is successfully leveraged by humans to handle novel situations. In recent years, transfer learning has received renewed interest from the community from different perspectives, including imitation learning, domain adaptation, and transfer of experience from simulation to the real world, among others. In this paper, we unify the concept of transfer learning in robotics and provide the first taxonomy of its kind considering the key concepts of robot, task, and environment. Through a review of the promises and challenges in the field, we identify the need of transferring at different abstraction levels, the need of quantifying the transfer gap and the quality of transfer, as well as the dangers of negative transfer. Via this position paper, we hope to channel the effort of the community towards the most significant roadblocks to realize the full potential of transfer learning in robotics.

Stable Jumping Control Based on Deep Reinforcement Learning for a Locust-Inspired Robot.

Biologically inspired jumping robots exhibit exceptional movement capabilities and can quickly overcome obstacles. However, the stability and accuracy of jumping movements are significantly compromised by rapid changes in posture. Here, we propose a stable jumping control algorithm for a locust-inspired jumping robot based on deep reinforcement learning. The algorithm utilizes a training framework comprising two neural network modules (actor network and critic network) to enhance training performance. The framework can control jumping by directly mapping the robot's observations (robot position and velocity, obstacle position, target position, etc.) to its joint torques. The control policy increases randomness and exploration by introducing an entropy term to the policy function. Moreover, we designed a stage incentive mechanism to adjust the reward function dynamically, thereby improving the robot's jumping stability and accuracy. We established a locus-inspired jumping robot platform and conducted a series of jumping experiments in simulation. The results indicate that the robot could perform smooth and non-flip jumps, with the error of the distance from the target remaining below 3%. The robot consumed 44.6% less energy to travel the same distance by jumping compared with walking. Additionally, the proposed algorithm exhibited a faster convergence rate and improved convergence effects compared with other classical algorithms.

A developmental model of audio-visual attention (MAVA) for bimodal language learning in infants and robots

A social individual needs to effectively manage the amount of complex information in his or her environment relative to his or her own purpose to obtain relevant information. This paper presents a neural architecture aiming to reproduce attention mechanisms (alerting/orienting/selecting) that are efficient in humans during audiovisual tasks in robots. We evaluated the system based on its ability to identify relevant sources of information on faces of subjects emitting vowels. We propose a developmental model of audio-visual attention (MAVA) combining Hebbian learning and a competition between saliency maps based on visual movement and audio energy. MAVA effectively combines bottom-up and top-down information to orient the system toward pertinent areas. The system has several advantages, including online and autonomous learning abilities, low computation time and robustness to environmental noise. MAVA outperforms other artificial models for detecting speech sources under various noise conditions.

What you need to know about a learning robot: Identifying the enabling architecture of complex systems

Nowadays we deal with robots and AI more and more in our everyday life. However, their behavior is not always apparent to most lay users, especially in error situations. This can lead to misconceptions about the behavior of the technologies being used. This in turn can lead to misuse and rejection by users. Explanation, for example through transparency, can address these misconceptions. However, explaining the entire software or hardware would be confusing and overwhelming for users. Therefore, this paper focuses on the ‘enabling’ architecture. It describes those aspects of a robotic system that may need to be explained to enable someone to use the technology effectively. Furthermore, this paper deals with the ‘explanandum’, i.e. the corresponding misunderstandings or missing concepts of the enabling architecture that need to be clarified. Thus, we have developed and are presenting an approach to determine the ‘enabling’ architecture and the resulting ‘explanandum’ of complex technologies.

Autonomous Robot Navigation Using Adaptive Potential Field

This work presents a novel approach to mobile robot autonomous navigation, utilizing enhanced artificial potential fields in conjunction with path planning,obstacle avoidance and SLAM . In traditional adaptive potential fields, which integrate sensor data to enable robots to navigate safely and efficiently through various environments. The robot's perception of its surroundings, obtained through sensors like cameras, ultrasonic devices, is processed to create a dynamic potential field that reflects the spatial distribution of obstacles, goals, and other relevant features.We propose the design and implementation of an autonomous robot-based real-time vision-based system for detecting and tracking features in a structured environment ensuring robust obstacle avoidance and path planning. Keywords- Adaptive Potential Fields, Autonomous Navigation, Autonomous Systems, Machine Learning in Robotics, Motor driver, Obstacle Avoidance, Path Planning, Raspberry Pi, Real-time Adaptation, ROS (Robot Operating System), Sensor-based Navigation, Servo Motor, SLAM (Simultaneous Localization and Mapping),Ultrasonic Sensors

An active SLAM with multi-sensor fusion for snake robots based on deep reinforcement learning

Snake-like robots can imitate the movement patterns of animals in nature and enter the space that traditional robots cannot enter, which adapt to environments that humans cannot reach, and expand the field of human exploration. However, it is often challenging to realize autonomous navigation and simultaneously avoid obstacles under an unknown environment, that is, active SLAM (Simultaneous Localization and Mapping). This paper proposes an autonomous obstacle avoidance method combined with SLAM based on deep reinforcement learning for a wheeled snake robot by using a multi-sensor. Firstly, we design a modular wheeled snake robot structure with lightweight materials based on orthogonal joints and build a three-dimensional model of a snake robot in Gazebo. Secondly, the SLAM based on two-dimensional LiDAR and IMU is used to realize autonomous navigation under an unknown environment and detect obstacles. At the same time, a Deep Q-Learning-based path planning method of the snake robot is proposed to realize obstacles avoidance during navigation. Finally, simulation studies and experiments show that the designed snake-like robot can realize effective path planning and environmental mapping in environments with obstacles. The proposed active SLAM algorithm improves the success rate of snake-like robot path planning, has better obstacle avoidance ability for obstacles, and reduces the number of collisions compared with the traditional A* and the sampling-based RRT* algorithms.

Task-Driven Reinforcement Learning With Action Primitives for Long-Horizon Manipulation Skills.

It is an interesting open problem to enable robots to efficiently and effectively learn long-horizon manipulation skills. Motivated to augment robot learning via more effective exploration, this work develops task-driven reinforcement learning with action primitives (TRAPs), a new manipulation skill learning framework that augments standard reinforcement learning algorithms with formal methods and parameterized action space (PAS). In particular, TRAPs uses linear temporal logic (LTL) to specify complex manipulation skills. LTL progression, a semantics-preserving rewriting operation, is then used to decompose the training task at an abstract level, informs the robot about their current task progress, and guides them via reward functions. The PAS, a predefined library of heterogeneous action primitives, further improves the efficiency of robot exploration. We highlight that TRAPs augments the learning of manipulation skills in both learning efficiency and effectiveness (i.e., task constraints). Extensive empirical studies demonstrate that TRAPs outperforms most existing methods.

Enhancing Robot Learning with Transformer-based Morphology Modeling

Abstract The transformer model has made significant progress in various areas through large-scale training. In contrast, the traditional robot performs a single task, and there is an issue with migrating the strategic model. In this study, a Robot Morphology Learning (RML) method is proposed to enhance efficiency and generalization performance by learning multiple tasks in a transformer model. RML constructs the robot’s morphology as a graph and utilizes a graph neural network to handle graphs of arbitrary connections and sizes, addressing the disparity in state and action space dimensions. RML breaks through the limitation of non-migration of models, realizes efficient training, and improves the generalization performance of models, enabling quick adaptation to new tasks. Experimental results show that the proposed method outperforms previous methods in both multi-task learning and transfer learning experiments.