Manipulation Tasks Research Articles

Design and Control Co-Optimization for Dynamic Loco-Manipulation With a Robotic Arm on a Quadruped Robot

Abstract Research in quadrupedal robotics is transitioning to studies into loco-manipulation, featuring fully articulated robotic arms mounted atop these robots. Integrating such arms enhances the practical utility of legged robots, paving the way for expanded applications like industrial inspection and search and rescue. Existing literature commonly employs a six-degree-of-freedom (six-DoF) arm directly mounted to the robot, which inherently adds significant weight and reduces the available payload for manipulation tasks. Our study explores an optimized combination of arm configuration and control framework by strategically reducing the DoFs and leveraging the quadruped robot’s inherent agile mobility. We demonstrate that by minimizing the DoFs to just one, a range of canonical loco-manipulation tasks can still be accomplished. Some tasks even show improved performance with fewer robotic arm DoFs due to the higher torque motor used in the design, allowing more of the robot’s payload to be used for manipulation. We designed our optimized one-DoF robotic arm and the control framework and tested it on top of a Unitree Aliengo. Our design outperforms conventional six-DoF counterparts in lifting capacity, achieving an impressive 8 kg payload compared to the 2 kg maximum payload of industry-standard six-DoF robotic arms on the same quadruped platform.

Design and Optimization of an Origami Gripper for Versatile Grasping and Manipulation

Robotic grasping and manipulation demand the ability to handle a multitude of objects with different shapes, sizes, quantities, surface smoothness, vulnerability, and stiffness, which is challenging without prior knowledge about object properties. Herein, a novel origami‐inspired gripper for universal grasping is presented. The innovative structure seamlessly transforms a simple uniaxial pulling motion into a flexible and robust envelope or pinch grasp, enabling it to tackle various scenarios. The origami gripper offers distinctive advantages, including scalable and optimizable design, grasping compliance and robustness, providing material flexibility, and providing solutions to challenging manipulation tasks. The working principles of the origami gripper are characterized and analyzed. An optimization‐based inverse design method is presented to adjust gripper properties for various scenarios. Through comprehensive experimentation and evaluation, the gripper's capabilities to grasp various objects with a wide range of distinctive properties, including ultrasoft, slippery, granular, and multiple objects, which is a challenge for the existing robotic grippers, are demonstrated. The research holds promise for transformative applications in areas such as the food industry, waste handling, fine and fragile objects grasping, and environmental sampling.

Robot skill acquisition for precision assembly of flexible flat cable with force control

Abstract The flexible flat cable (FFC) assembly task is a prime challenge in electronic manufacturing. Its characteristics of being prone to deformation under external force, tiny assembly tolerance, and fragility impede the application of robotic assembly in this field. To achieve reliable and stable robotic automation assembly of FFC, an efficient assembly skill acquisition strategy is presented by combining a parallel robot skill learning algorithm with adaptive impedance control. The parallel robot skill learning algorithm is proposed to enhance the efficiency of FFC assembly skill acquisition, which reduces the risk of damaging FFC and tackles the uncertain influence resulting from deformation during the assembly process. Moreover, FFC assembly is also a complex contact-rich manipulation task. An adaptive impedance controller is designed to implement force tracking during the assembly process without precise environment information, and the stability is also analyzed based on the Lyapunov function. Experiments of FFC assembly are conducted to illustrate the efficiency of the proposed method. The experimental results demonstrate that the proposed method is robust and efficient.

Hey Building! Novel Interfaces for Parametric Design Manipulations in Virtual Reality

Parametric Design enables designers to formulate and explore new ideas through parameters, typically by manipulating numerical values. However, visualising and exploring the design space of an established parametric design solution is natively difficult through desktop displays for designers due to screen space constraints and requiring familiarity with visual-language programming interfaces. Thus, we sought to explore Virtual Reality (VR), inspired by Natural User Interfaces (NUI), to develop and explore new interfaces departing from traditional programming interfaces, that could complement the spatial and embodied affordances of contemporary VR devices. Informed by two industry-led focus groups with architects we developed and examined the usability of three different interfaces: 1) Paramaxes , an axes-based interface that allows designers to distribute and manipulate parameter visualisations around them in physical space; 2) ParamUtter , a Voice-based User Interface (VUI) that allows designers to manipulate parameter visualisations through natural languages; 3) Control Panel , which presents the parameters as sliders in a scrollable pane and acts as baseline comparison. We ran an exploratory study with experts and found that the Control Panel was ultimately the preferred interface for a design manipulation task. However, participants commented favorably towards qualities in the unconventional interfaces, with ParamUtter scoring highest in System Usability Scores (SUS), and participants valuing the potential of using physical space to explore design spaces with Paramaxes .

Robot learning on the job: Human-in-the-loop autonomy and learning during deployment

With the rapid growth of computing powers and recent advances in deep learning, we have witnessed impressive demonstrations of novel robot capabilities in research settings. Nonetheless, these learning systems exhibit brittle generalization and require excessive training data for practical tasks. To harness the capabilities of state-of-the-art robot learning models while embracing their imperfections, we present Sirius, a principled framework for humans and robots to collaborate through a division of work. In this framework, partially autonomous robots are tasked with handling a major portion of decision-making where they work reliably; meanwhile, human operators monitor the process and intervene in challenging situations. Such a human–robot team ensures safe deployments in complex tasks. Further, we introduce a new learning algorithm to improve the policy’s performance on the data collected from the task executions. The core idea is re-weighing training samples with approximated human trust and optimizing the policies with weighted behavioral cloning. We evaluate Sirius in simulation and on real hardware, showing that Sirius consistently outperforms baselines over a collection of contact-rich manipulation tasks, achieving an 8% boost in simulation and 27% on real hardware than the state-of-the-art methods in policy success rate, with twice faster convergence and 85% memory size reduction. Videos and more details are available at https://ut-austin-rpl.github.io/sirius/ .

Practical Probabilistic Model-Based Reinforcement Learning by Integrating Dropout Uncertainty and Trajectory Sampling.

This article addresses the prediction stability, prediction accuracy, and control capability of the current probabilistic model-based reinforcement learning (MBRL) built on neural networks. A novel approach to dropout-based probabilistic ensembles with trajectory sampling (DPETS) is proposed, where the system uncertainty is stably predicted by combining the Monte Carlo dropout (MC Dropout) and trajectory sampling in one framework. Its loss function is designed to correct the fitting error of neural networks for more accurate prediction of probabilistic models. The state propagation in its policy is extended to filter the aleatoric uncertainty for superior control capability. Evaluated by several Mujoco benchmark control tasks under additional disturbances and one practical robot arm manipulation task, DPETS outperforms related MBRL approaches in both average return and convergence velocity while achieving superior performance than well-known model-free baselines with significant sample efficiency. The open-source code of DPETS is available at https://github.com/mrjun123/DPETS.

Sex differences in autonomic functions and cognitive performance during cold-air exposure and cold-water partial immersion.

This study investigated the differences between males and females in autonomic functions and cognitive performance during cold-air exposure and cold-water partial-immersion compared to a room temperature-air environment. Although several studies have investigated the effects of cold-air or cold-water exposures on autonomic function and cognitive performance, biological sex differences are often under-researched. Twenty-two males and nineteen females participated in the current study. Subjects completed a battery of cognitive tasks based upon those used within the Defense Automated Neurobehavioral Assessment (DANA), consisting of five subtasks that assess simple and procedural reaction time, spatial manipulation, attention, and immediate memory. In total, subjects took the battery within a 15-minute period across 30-minute intervals throughout the duration of environmental exposure. Across three separate days, subjects were exposed to three different environmental conditions: room temperature air (23°C), cold air (10°C), and cold water (15°C; in which subjects were immersed up to their necks). Room temperature and cold-air conditions consisted of five sessions (about 2.5h), and the cold-water condition consisted of three sessions (about 1.5h). During each experimental condition, physiological data were collected to assess autonomic function, including electrodermal activity (EDA) data and heart rate variability (HRV) derived from electrocardiogram signals. Females showed slower reaction time in spatial manipulation tasks, immediate memory, and attention during cold-air exposures compared to room temperature air, whereas the performance of males were similar or better during cold-air exposures compared to room temperature air. Cold-water immersion affected the immediate memory performance of males. Both males and females exhibited smaller EDA amplitudes during cold-air and cold-water conditions compared to room temperature air. For HRV, only male subjects exhibited significantly greater values in low-frequency and very-low-frequency components during cold air exposure compared to the normal condition. Sex introduces important differences in cognitive performance and autonomic functions during exposure to cold-air and cold-water. Therefore, sex should be considered when assessing the autonomic nervous system in cold environments and when establishing optimal thermal clothing for performance in operational environments. Our findings can assist with determination of operational clothing, temperature in operating environment, and personnel deployment to operational sites, particularly in settings involving both males and females.

Assessing motivational biases in brain and behavior: Event-related potential and response time concomitants of the approach-avoidance task.

The approach-avoidance task (AAT) is designed to measure implicit motivated action biases instantiated by emotional stimuli and alterations in such biases that drive psychiatric disorder. While some research has measured AAT event-related potential (ERP) correlates to establish bias sensitivity even at a neural level, a lack of work with unpleasant, pleasant, and neutral stimuli together and a common focus on psychiatric disorder-matched (rather than generally emotional) content limits conclusions that can be drawn. Thus, current work extends the AAT literature by testing ERP modulations across normatively unpleasant, pleasant, and neutral conditions; and supporting the task's use as an individual difference assessment, it also provides data on AAT reliability and initially explores anxiety-related effects when stimuli are not disorder-matched. In 38 participants including 19 anxiety treatment-seeking individuals, 32 sensor electroencephalography revealed robust N100, N200, and late positive potential (LPP) ERP components and bias-consistent modulations for unpleasant images (reduced N200s on unpleasant push relative to pull trials; enhanced LPP for unpleasant compared to neutral trials). Meanwhile, modulations were less consistent with emotion-driven bias for other conditions-that is, LPPs were enhanced but N200 was not modulated for pleasant images, and for neutral images, N200 was unexpectedly enhanced on push compared to pull trials. Following these analyses, reliability tests revealed excellent raw ERP reliabilities but lower reliabilities for modulation scores, and comparing treatment- to non-treatment-seeking groups showed no preliminary indication of ERP modulation changes when stimuli are not personally relevant. How these findings together inform understanding of AAT as a measure of bias is discussed.

Reinforcement Learning with Decoupled State Representation for Robot Manipulations

Deep reinforcement learning has significantly advanced robot manipulations by providing an alternative solution for designing control strategies using raw images as direct inputs. While images offer additional environmental information, the end-to-end policy training manner (from image to action) requires simultaneous representation and task learning by the agent. This often necessitates a substantial number of interaction samples to achieve satisfactory policy performance. Previous works has attempted to address this challenge by learning a visual representation model that encodes the entire image into a low-dimensional vector before the policy training. However, since this vector contains both robot and object information, it inevitably introduces coupling within the state, which can mislead the policy training process. In this study, a novel method called Reinforcement Learning with Decoupled State Representation is proposed to effectively decouple robot and object information within the state representation. Experimental results demonstrate that the proposed method exhibits faster learning speed and achieves superior performance compared to previous methods across various robot manipulation tasks. Moreover, with only 3096 offline images, the proposed method successfully applies to real-world robot pushing tasks, which demonstrates its high practicability.

Review of Flexible Robotic Grippers, with a Focus on Grippers Based on Magnetorheological Materials

Flexible grippers are a promising and pivotal technology for robotic grasping and manipulation tasks. Remarkably, magnetorheological (MR) materials, recognized as intelligent materials with exceptional performance, are extensively employed in flexible grippers. This review aims to provide an overview of flexible robotic grippers and highlight the application of MR materials within them, thereby fostering research and development in this field. This work begins by introducing various common types of flexible grippers, including shape memory alloys (SMAs), pneumatic flexible grippers, and dielectric elastomers, illustrating their distinctive characteristics and application domains. Additionally, it explores the development and prospects of magnetorheological materials, recognizing their significant contributions to the field. Subsequently, MR flexible grippers are categorized into three types: those with viscosity/stiffness variation capabilities, magnetic actuation systems, and adhesion mechanisms. Each category is comprehensively analyzed, specifying its unique features, advantages, and current cutting-edge applications. By undertaking an in-depth examination of diverse flexible robotic gripper types and the characteristics and application scenarios of MR materials, this paper offers a valuable reference for fellow researchers. As a result, it facilitates further advancements in this field and contributes to the provision of efficient gripping solutions for industrial automation.

Learning-Based Slip Detection for Dexterous Manipulation Using GelStereo Sensing.

Endowing the robot with tactile perception can effectively improve manipulation dexterity, along with various benefits of human-like touch. Using GelStereo (GS) tactile sensing, which gives high-resolution contact geometry information, including 2-D displacement field, and 3-D point cloud of the contact surface, we present a learning-based slip detection system in this study. The results reveal that the well-trained network achieves 95.79% accuracy on the never-seen testing dataset, which surpasses the current model-based and learning-based methods using visuotactile sensing. We also propose a general framework for slip feedback adaptive control for dexterous robot manipulation tasks. The experimental results show the effectiveness and efficiency of the proposed control framework using GS tactile feedback when deployed on real-world grasping and screwing manipulation tasks on various robot setups.

Topology, dynamics, and control of a muscle-architected soft arm

Muscular hydrostats, such as octopus arms or elephant trunks, lack bones entirely, endowing them with exceptional dexterity and reconfigurability. Key to their unmatched ability to control nearly infinite degrees of freedom is the architecture into which muscle fibers are weaved. Their arrangement is, effectively, the instantiation of a sophisticated mechanical program that mediates, and likely facilitates, the control and realization of complex, dynamic morphological reconfigurations. Here, by combining medical imaging, biomechanical data, live behavioral experiments, and numerical simulations, an octopus-inspired arm made of [Formula: see text]200 continuous muscle groups is synthesized, exposing "mechanically intelligent" design and control principles broadly pertinent to dynamics and robotics. Such principles are mathematically understood in terms of storage, transport, and conversion of topological quantities, effected into complex 3D motions via simple muscle activation templates. These are in turn composed into higher-level control strategies that, compounded by the arm's compliance, are demonstrated across challenging manipulation tasks, revealing surprising simplicity and robustness.

Enhanced Impedance Control of Cable-Driven Unmanned Aerial Manipulators Using Fractional-Order Nonsingular Terminal Sliding Mode Control with Disturbance Observer Integration

The article presents a novel control strategy for cable-driven aerial manipulators (UAMs) aimed at enhancing impedance control during contact operations in complex environments. A fractional-order nonsingular terminal sliding mode control (FONTSMC) integrated with a disturbance observer (DOB) is proposed to improve the robustness and precision of the UAM under lumped disturbances. This developed approach utilizes the flexibility of fractional calculus, the finite-time stability of nonsingular terminal sliding mode, and the real-time disturbance estimation capabilities of the DOB to ensure smooth and compliant contact interactions. The effectiveness of the proposed control strategy is validated through comprehensive simulation studies, which demonstrate significant improvements in control performance, stability, and disturbance rejection when compared to traditional methods. The results indicate that the FONTSMC-DOB framework is highly suitable for complex aerial manipulation tasks, offering both theoretical and practical insights into the design of advanced control systems for UAMs.

On Multi-Fidelity Impedance Tuning for Human-Robot Cooperative Manipulation

Abstract We examine how a human-robot interaction (HRI) system may be designed when input output data from previous experiments are available. Our objective is to learn an optimal impedance in the assistance design for a cooperative manipulation task with a new operator. Due to the variability between individuals, the design parameters that best suit one operator of the robot may not be the best parameters for another one. However, by incorporating historical data using a linear auto-regressive (AR-1) Gaussian process, the search for a new operator's optimal parameters can be accelerated. We lay out a framework for optimizing the human-robot cooperative manipulation that only requires input-output data. We characterize the learning performance using a notion called regret, establish how the AR-1 model improves the bound on the regret, and numerically illustrate this improvement in the context of a human-robot cooperative manipulation task. Further, we show how our approach's input-output nature provides robustness against modeling error through an additional numerical study.

Method for Bottle Opening with a Dual-Arm Robot.

This paper introduces a novel approach to robotic assistance in bottle opening using the dual-arm robot TIAGo++. The solution enhances accessibility by addressing the needs of individuals with injuries or disabilities who may require help with common manipulation tasks. The aim of this paper is to propose a method involving vision, manipulation, and learning techniques to effectively address the task of bottle opening. The process begins with the acquisition of bottle and cap positions using an RGB-D camera and computer vision. Subsequently, the robot picks the bottle with one gripper and grips the cap with the other, each by planning safe trajectories. Then, the opening procedure is executed via a position and force control scheme that ensures both grippers follow the unscrewing path defined by the cap thread. Within the control loop, force sensor information is employed to control the vertical axis movements, while gripper rotation control is achieved through a Deep Reinforcement Learning (DRL) algorithm trained to determine the optimal angle increments for rotation. The results demonstrate the successful training of the learning agent. The experiments confirm the effectiveness of the proposed method in bottle opening with the TIAGo++ robot, showcasing the practical viability of the approach.

FMB: A functional manipulation benchmark for generalizable robotic learning

In this paper, we propose a real-world benchmark for studying robotic learning in the context of functional manipulation: a robot needs to accomplish complex long-horizon behaviors by composing individual manipulation skills in functionally relevant ways. The core design principles of our Functional Manipulation Benchmark (FMB) emphasize a harmonious balance between complexity and accessibility. Tasks are deliberately scoped to be narrow, ensuring that models and datasets of manageable scale can be utilized effectively to track progress. Simultaneously, they are diverse enough to pose a significant generalization challenge. Furthermore, the benchmark is designed to be easily replicable, encompassing all essential hardware and software components. To achieve this goal, FMB consists of a variety of 3D-printed objects designed for easy and accurate replication by other researchers. The objects are procedurally generated, providing a principled framework to study generalization in a controlled fashion. We focus on fundamental manipulation skills, including grasping, repositioning, and a range of assembly behaviors. The FMB can be used to evaluate methods for acquiring individual skills, as well as methods for effectively combining and ordering such skills in order to solve complex, multi-stage manipulation tasks. We also offer an imitation learning framework that includes a suite of policies trained to solve the proposed tasks. This enables researchers to utilize our tasks as a versatile toolkit for examining various parts of the pipeline. For example, researchers could propose a better design for a grasping controller and evaluate it in combination with our baseline reorientation and assembly policies as part of a pipeline for solving multi-stage tasks. Our dataset, object CAD files, code, and evaluation videos can be found on our project website: https://functional-manipulation-benchmark.github.io .

Mechanochromic Suction Cups for Local Stress Detection in Soft Robotics

Advancements in smart soft materials are enhancing the capabilities of robotic manipulators in object interactions and complex tasks. Mechanochromic materials, acting as lightweight sensors, offer easily interpretable visual feedback for localized stress detection, structural health monitoring, and energy‐efficient robotic skins. Herein, an innovative mechanochromic soft end‐effector capable of discerning local contact stresses during mechanical interactions with objects is presented and their relative position is ascertained. This system utilizes a reversible force‐induced color switch in a thin layer of spiropyran‐functionalized polydimethylsiloxane, which coats a silicone‐made suction cup. The mechanochromic suction cup is integrated with a 3D‐printed compact load‐transferring system and electronic color‐changing detection elements. The assembly may serve as a synthetic receptor for robotic actuators, discerning localized interaction forces down to 3 N. The system's resilience to varying environmental factors, including illumination, tilting, and interaction with objects of various shapes is verified. The results indicate potential for exteroceptive solutions in reconfigurable manipulation tasks without compromising the overall softness of the manipulator.

Dynamic modeling and trajectory optimization for the rigid-flexible coupled spacecraft with the free-floating manipulator and solar panels

The rigid-flexible coupled spacecraft, composed of flexible solar panels and a multilink manipulator, has gained prominence in on-orbit servicing due to rapid advancements in space technology. However, the intricate effects of rigid-flexible coupling pose significant challenges for dynamic modeling, trajectory planning, and control. This paper aims to develop general dynamic approaches for modeling and trajectory planning in such spacecraft, considering large deformations. The main distinguishing feature is the use of the referenced nodal coordinate formulation to accurately describe the large-deformed solar panels rather than directly treating them as disturbance for the free-floating system. Additionally, the common recursive model for the multilink manipulator is integrated into the same framework. The modal reduction method with modal derivatives techniques is employed to address geometric nonlinearity resulting from large deformations. Polynomial trajectory parameters with different performance characteristics are obtained by defining various optimization objectives. The coupling analysis is conducted based on an accurate reduced-order dynamic model, the results of which can be used for designing manipulator tasks. Coupling values are defined as the objective for trajectory optimization, offering advantages such as insensitivity to dynamic model accuracy and a fast optimization process. After validating the accuracy of the proposed dynamic model through simulations, the performances of different optimized trajectories are demonstrated using numerical examples.

OneTip: A Soft Tactile Interface for 6-D Fingertip Pose Acquisition in Human-Computer Interaction

Advances in display technology have created the need for more efficient and natural multi-degree-of-freedom interaction devices. The movement of a single fingertip has six degrees of freedom (DOFs), but traditional rigid touchscreens usually sense only 2-D information. This article proposes a new deformable tactile interface, OneTip, for single-fingertip human-computer interaction with 6 DOFs. It is manufactured based on the principle of vision-based tactile sensing using virtual stereoscopic cameras, and its size is about twice that of a thumb. The contact surface of OneTip has a specially designed structure and material to mimic the sensitivity and softness of human skin. Also, OneTip employs a new sensing method to address the problem of soft fingertip pose estimation (measuring relative change only) with incipient slip effects. Experiments show that OneTip has good 6-D pose estimation accuracy, with root mean square errors (RMSE) of translation and rotation not exceeding 0.1mm and 2.6°, respectively, within the linear interval (x and y: −1.2 to 1.2mm; z: 0 to 3mm; yaw: −15 to 15deg; pitch and roll: −40 to 40deg). Experiments were also conducted to explore the application of OneTip in typical virtual manipulation tasks and the possibility of combining it with other interaction devices.

Learning human actions from complex manipulation tasks and their transfer to robots in the circular factory

Abstract Process automation is essential to establish an economically viable circular factory in high-wage locations. This involves using autonomous production technologies, such as robots, to disassemble, reprocess, and reassemble used products with unknown conditions into the original or a new generation of products. This is a complex and highly dynamic issue that involves a high degree of uncertainty. To adapt robots to these conditions, learning from humans is necessary. Humans are the most flexible resource in the circular factory and they can adapt their knowledge and skills to new tasks and changing conditions. This paper presents an interdisciplinary research framework for learning human action knowledge from complex manipulation tasks through human observation and demonstration. The acquired knowledge will be described in a machine-executable form and will be transferred to industrial automation execution by robots in a circular factory. There are two primary research objectives. First, we investigate the multi-modal capture of human behavior and the description of human action knowledge. Second, the reproduction and generalization of learned actions, such as disassembly and assembly actions on robots is studied.