Robot Execution Research Articles

Modified Deng’s Grey Relational Analysis Model for Panel Data and Its Applications in Assessing the Water Environment of Poyang Lake

Deng’s grey relational analysis (GRA) model is widely used in clustering because of its simple mathematical mechanisms. For sample data of different dimensions, people have put forward different Deng’s GRA models, including time series data, panel data, and panel time series data. The purpose of this paper is to improve the clustering accuracy of the existing Deng’s GRA model for panel data in order to overcome some of its shortcomings. Firstly, the existing Deng’s GRA model for panel data was tested based on the dataset LP1 of Robot Execution Failures. Then, according to the test results, the existing Deng’s GRA model for panel data is modified by means of Taylor’s formula, and the modified model is successfully validated by the dataset LP1 of Robot Execution Failures. Finally, as a practical application, the modified Deng’s GRA model for panel data is applied to assess the water environment of Poyang Lake over the past five years. Compared with other cluster methods, the results of the case study show that the modified Deng’s GRA model for panel data is applicable and also confirm the remarkable effectiveness of the Chinese government’s water quality regulation in Poyang Lake. Therefore, the modified Deng’s GRA model presented in this paper improves the clustering accuracy compared to the original model and can be applied well to the classification of data with a large dimension.

A Review of Natural-Language-Instructed Robot Execution Systems

A Review of Natural-Language-Instructed Robot Execution Systems

Keep the Human in the Loop: Arguments for Human Assistance in the Synthesis of Simulation Data for Robot Training

Robot training often takes place in simulated environments, particularly with reinforcement learning. Therefore, multiple training environments are generated using domain randomization to ensure transferability to real-world applications and compensate for unknown real-world states. We propose improving domain randomization by involving human application experts in various stages of the training process. Experts can provide valuable judgments on simulation realism, identify missing properties, and verify robot execution. Our human-in-the-loop workflow describes how they can enhance the process in five stages: validating and improving real-world scans, correcting virtual representations, specifying application-specific object properties, verifying and influencing simulation environment generation, and verifying robot training. We outline examples and highlight research opportunities. Furthermore, we present a case study in which we implemented different prototypes, demonstrating the potential of human experts in the given stages. Our early insights indicate that human input can benefit robot training at different stages.

RTMN 2.0—An Extension of Robot Task Modeling and Notation (RTMN) Focused on Human–Robot Collaboration

This paper describes RTMN 2.0, an extension of the modeling language RTMN. RTMN combines process modeling and robot execution. Intuitive robot programming allows those without programming expertise to plan and control robots through easily understandable predefined modeling notations. These notations achieve no-code programming and serve as templates for users to create their processes via drag-and-drop functions with graphical representations. The design of the graphical user interface is based on a user survey and gaps identified in the literature We validate our survey through the most influential technology acceptance models, with two major factors: the perceived ease of use and perceived usefulness. While RTMN focuses on the ease of use and flexibility of robot programming by providing an intuitive modeling language, RTMN 2.0 concentrates on human–robot collaboration (HRC), which represents the current trend of the industry shift from “mass-production” to “mass-customization”. The biggest contribution that RTMN 2.0 makes is the creation of synergy between HRC modes (based on ISO standards) and HRC task types in the literature. They are modeled as five different HRC task notations: Coexistence Fence, Sequential Cooperation SMS, Teaching HG, Parallel Cooperation SSM, and Collaboration PFL. Both collaboration and safety criteria are defined for each notation. While traditional isolated robot systems in “mass-production” environments provide high payload capabilities and repeatability, they suffer from limited flexibility and dexterity in order to be adapted to the variability of customized products. Therefore, human–robot collaboration is a suitable arrangement to leverage the unique capabilities of both humans and robots for increased efficiency and quality in the new “mass-customization” industrial environments. HRC has made a great impact on the robotic industry: it leads to increased efficiency, reduced costs, and improved productivity, which can be adopted to make up for the skill gap of a shortage of workers in the manufacturing industry. The extension of RTMN 2.0 includes the following notations: HRC tasks, requirements, Key Performance Indicators (KPIs), condition checks and decision making, join/split, and data association. With these additional elements, RTMN 2.0 meets the full range of criteria for agile manufacturing—light-out manufacturing is a manufacturing philosophy that does not rely on human labor.

ASAP-CORPS: A Semi-Autonomous Platform for COntact-Rich Precision Surgery.

Remote military operations require rapid response times for effective relief and critical care. Yet, the military theater is under austere conditions, so communication links are unreliable and subject to physical and virtual attacks and degradation at unpredictable times. Immediate medical care at these austere locations requires semi-autonomous teleoperated systems, which enable the completion of medical procedures even under interrupted networks while isolating the medics from the dangers of the battlefield. However, to achieve autonomy for complex surgical and critical care procedures, robots require extensive programming or massive libraries of surgical skill demonstrations to learn effective policies using machine learning algorithms. Although such datasets are achievable for simple tasks, providing a large number of demonstrations for surgical maneuvers is not practical. This article presents a method for learning from demonstration, combining knowledge from demonstrations to eliminate reward shaping in reinforcement learning (RL). In addition to reducing the data required for training, the self-supervised nature of RL, in conjunction with expert knowledge-driven rewards, produces more generalizable policies tolerant to dynamic environment changes. A multimodal representation for interaction enables learning complex contact-rich surgical maneuvers. The effectiveness of the approach is shown using the cricothyroidotomy task, as it is a standard procedure seen in critical care to open the airway. In addition, we also provide a method for segmenting the teleoperator's demonstration into subtasks and classifying the subtasks using sequence modeling. A database of demonstrations for the cricothyroidotomy task was collected, comprising six fundamental maneuvers referred to as surgemes. The dataset was collected by teleoperating a collaborative robotic platform-SuperBaxter, with modified surgical grippers. Then, two learning models are developed for processing the dataset-one for automatic segmentation of the task demonstrations into a sequence of surgemes and the second for classifying each segment into labeled surgemes. Finally, a multimodal off-policy RL with rewards learned from demonstrations was developed to learn the surgeme execution from these demonstrations. The task segmentation model has an accuracy of 98.2%. The surgeme classification model using the proposed interaction features achieved a classification accuracy of 96.25% averaged across all surgemes compared to 87.08% without these features and 85.4% using a support vector machine classifier. Finally, the robot execution achieved a task success rate of 93.5% compared to baselines of behavioral cloning (78.3%) and a twin-delayed deep deterministic policy gradient with shaped rewards (82.6%). Results indicate that the proposed interaction features for the segmentation and classification of surgical tasks improve classification accuracy. The proposed method for learning surgemes from demonstrations exceeds popular methods for skill learning. The effectiveness of the proposed approach demonstrates the potential for future remote telemedicine on battlefields.

Feature-level SMOTE: Augmenting fault samples in learnable feature space for imbalanced fault diagnosis of gas turbines

A challenge in gas turbine fault diagnosis is that labeled fault samples are relatively rare and much fewer than normal samples. Conventional data augmentation techniques generate fault samples in original data spaces, resulting in the issue that synthetic fault samples highly overlap with normal samples. Aiming at the issue, a feature-level data augmentation method, namely feature-level SMOTE, is developed by integrating deep Siamese multi-head self-attention network (DSMHSA) with synthetic minority over-sampling technique (SMOTE) to reduce inter-class imbalance and overlap simultaneously. First, the DSMHSA maps original data into a feature space with better inter-class separability, in which inter-class samples stay far away from one another. Second, the SMOTE generates synthetic fault samples in the well-separable space, in order to balance the data set. Finally, the effectiveness of the developed feature-level SMOTE in imbalanced fault diagnosis has been evaluated through two case studies including the real gas turbine fault dataset and the public robot execution failures dataset. To be specific, its average balanced accuracy is 90.38% on the gas turbine dataset, yielding 9.67%, 13.94%, and 12.39% improvements compared to the OUPS, A-SUWO, and NRAS, respectively.

Offline Time-Independent Multiagent Path Planning

This study examines a novel planning problem for multiple agents that cannot share holding resources, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Offline Time-Independent Multiagent Path Planning (OTIMAPP)</i> . Given a graph and a set of start-goal pairs, the problem to be addressed is assigning a path to each agent, such that every agent eventually reaches its destination without blocking others, regardless of when each agent starts and finishes each own action. This motivation stems from timing uncertainties, including the reality gaps between planning and robot execution. In contrast to conventional solution, concepts of multirobot path planning that rely on timings, once OTIMAPP solutions are obtained, they can be executed without any synchronization between robot actions. Moreover, there is a theoretical guarantee that all robots eventually reach their destinations, provided they avoid interrobot collisions. This study attempts to establish OTIMAPP both theoretically and practically. Specifically, we present a formalization of the problem, solution conditions based on a categorization of deadlocks, computational complexities showing that OTIMAPP is computationally intractable, practical relaxation of the solution concept, two algorithms to solve OTIMAPP based on multiagent pathfinding algorithms, empirical results showing large OTIMAPP instances can be solved to some extent, as well as robot demonstrations of asynchronous OTIMAPP execution.

HOW MUCH CAN A SURGEON TRUST A ROBOT FOR HIP ARTHROPLASTY?

The variables involved in a robotic THA can exceed 52: many parameters as pelvic orientation with CT scan, templating, offset, and leg-length, acetabular reaming, femoral osteotomy, mapping the anatomy; predefining safe zones, robotic execution, femoral head size, thickness of PE etc. with several variables for each parameter, with a total number of variables exceeding 52. This familiar number is the number of cards in a standard deck. The number of possible combinations (factorial 52! = 10^67) to shuffle the cards (and may be to perform a THA) is greater than the number of atoms on earth! Thinking that artificial intelligence and robotics can solve these problems, some surgeons and implant manufacturers have turned to artificial intelligence and robotics.We asked two questions:1) can robot with artificial intelligence really process 52 variables that represent 10^67 combinations? 2) the safety of the technology was ascertained by interrogating Food and Drug Administration (FDA) database about software-related recalls in computer-assisted and robotic arthroplasty [1], between 2017 and 2022.1)The best computers can only calculate around 100 thousand billion combinations (10^14), and with difficulty: it takes more than 100 days to arrive at this number of digits (10^14) after the decimal point for the number π (pi). We can, therefore, expect the robot to be imperfect.2)For the FDA software-related recalls, 4634 units were involved. The FDA determined root causes were: software design (66.6%), design change (22.2%), manufacturing deployment (5.6%), design manufacturing process (5.6%). Among the manufacturers’ reasons for recalls, a specific error was declared in 88.9%. a coding error in 43.8%. 94.4% software-related recalls were classified as class 2. Return of the device was the main action taken by firms (44.4%), followed by software update (38.9%).3)In the same period, no robot complained about its surgeon!Hip surgeon is as intelligent as a robot and almost twice as safe.

Robot learning from human demonstrations with inconsistent contexts

Visual imitation learning is a promising approach that promotes robots to learn skills from visual demonstrations. However, current visual imitation learning approaches introduce unreasonable assumptions that the contexts of the visual demonstrations and the robot observations are consistent, which affects the flexibility and scalability of the approaches. It is a key challenge for robots to learn from visual demonstrations with inconsistent contexts. Inconsistent contexts may cause a serious difference in the pixel distribution of the operator and the environment, which makes vision-based control policies hardly effective. In this paper, we propose a novel imitation learning framework to enable robots to reproduce behavior by watching human demonstrations with inconsistent contexts, such as different viewpoints, operators, backgrounds, object appearances and positions. Specifically, our framework consists of three networks: flow-based viewpoint transformation network (FVTrans), robot2human alignment network (RANet) and inverse dynamics network (IDNet). First, FVTrans transforms various third-person demonstrations into the fixed robot execution view. With a meta learning strategy, FVTrans can quickly adapt to novel contexts with few samples. Then, RANet aligns the human and the robot at the feature level. Therefore, the demonstration feature can be used as a subgoal of the current moment. Finally, IDNet predicts the joint angles of the robot. We collect a multi-context dataset on the real robot (UR5) for three tasks, including grasping cups, sweeping garbage and placing objects. We empirically demonstrate that our framework can perform three tasks with a high success rate and be effectively generalized to different contexts.

Learning From Sparse Demonstrations

In this article, we develop the method of continuous Pontryagin differentiable programming (Continuous PDP), which enables a robot to learn an objective function from a few sparsely demonstrated keyframes. The keyframes, labeled with some time stamps, are the desired task-space outputs, which a robot is expected to follow sequentially. The time stamps of the keyframes can be different from the time of the robot's actual execution. The method jointly finds an objective function and a time-warping function such that the robot's resulting trajectory sequentially follows the keyframes with minimal discrepancy loss. The Continuous PDP minimizes the discrepancy loss using projected gradient descent by efficiently solving the gradient of the robot trajectory with respect to the unknown parameters. The method is first evaluated on a simulated robot arm and then applied to a 6-DoF quadrotor to learn an objective function for motion planning in unmodeled environments. The results show the efficiency of the method, its ability to handle time misalignment between keyframes and robot execution, and the generalization of objective learning into unseen motion conditions.

ASAP: A Semi-Autonomous Precise System for Telesurgery During Communication Delays

In remote, rural, and disadvantaged areas, telesurgery can be severely hindered by limitations of communication infrastructure. In conventional telesurgery, delays as small as 300ms can produce fatal surgical errors. To mitigate the effect of communication delays during telesurgery, we introduce a semi-autonomous system that decouples the user interaction from the robot execution. This system uses a physics-based simulator where a surgeon can demonstrate individual surgical subtasks, with immediate graphical feedback. Each subtask is performed asynchronously, unaffected by communication latency, jitter, and packet loss. A surgical step recognition module extracts the intended actions from the observed surgeon-simulation interaction. The remote robot can perform each one of these actions autonomously. The action recognition system leveraged a transfer learning approach that minimized the data needed during training, and most of the learning is obtained from simulated data. We tested this system in two tasks: fluid-submerged peg transfer (resembling bleeding events) and surgical debridement. The system showed robustness to delays of up to 5 seconds, maintaining a performance rate of 87% for peg transfer and 88% for debridement. Also, the framework reduced the completion time under delays by 45% and 11% during peg transfer and debridement, respectively.

Experimental temperature evaluation during a robotized bone drilling process

A serious problem associated with orthopedic bone drilling manipulation is the so-called thermal osteonecrosis (TO). This problem occurs because the drilling manipulation generates heat, and the amount of heat depends on the drilling conditions. This work presents the experimental evaluation of the temperature during automated bone drilling by using a handheld Orthopedic bone Drilling Robot ODRO. Up to now, such experimental results related to temperature evaluation in bone drilling by robots have not been presented. Fresh porcine and bovine femur bones were specimens in the experiments. The increase in bone temperature during drilling was measured by a thermal infrared camera. Thermographic sequence images were recorded, processed and visualized using specialized software, which allows to capture accurately the moment of drill bit penetration through the bone and its extraction when going back to initial position. The results showed that under given feed rate and drill speed, when using a new drill bit, the temperature value during porcine femur drilling exceeded the critical threshold of 50 °C for less than 2 s, and thermal osteonecrosis does not occur. When using a worn-out drill bit, the temperature value during bovine femur drilling significantly exceeded the threshold of 50 °C, and thermal osteonecrosis occurs. Thus, to guarantee bone drilling without the risk of thermal osteonecrosis, the automated drilling should be executed with the simultaneous control of speed of drill bit and feed rate. This is possible to be achieved only by robotic execution of the manipulation.

Computer-vision-guided semi-autonomous concrete crack repair for infrastructure maintenance using a robotic arm

Engineering inspection and maintenance technologies play an important role in safety, operation, maintenance and management of buildings. In project construction control, supervision of engineering quality is a difficult task. To address such inspection and maintenance issues, this study presents a computer-vision-guided semi-autonomous robotic system for identification and repair of concrete cracks, and humans can make repair plans for this system. Concrete cracks are characterized through computer vision, and a crack feature database is established. Furthermore, a trajectory generation and coordinate transformation method is designed to determine the robotic execution coordinates. In addition, a knowledge base repair method is examined to make appropriate decisions on repair technology for concrete cracks, and a robotic arm is designed for crack repair. Finally, simulations and experiments are conducted, proving the feasibility of the repair method proposed. The result of this study can potentially improve the performance of on-site automatic concrete crack repair, while addressing such issues as high accident rate, low efficiency, and big loss of skilled workers.

A hybrid skill parameterisation model combining symbolic and subsymbolic elements for introspective robots

In the design of robot skills, the focus generally lies on increasing the flexibility and reliability of the robot execution process; however, typical skill representations are not designed for analysing execution failures if they occur or for explicitly learning from failures. In this paper, we describe a learning-based hybrid representation for skill parameterisation called an execution model, which considers execution failures to be a natural part of the execution process. We then (i) demonstrate how execution contexts can be included in execution models, (ii) introduce a technique for generalising models between object categories by combining generalisation attempts performed by a robot with knowledge about object similarities represented in an ontology, and (iii) describe a procedure that uses an execution model for identifying a likely hypothesis of a parameterisation failure. The feasibility of the proposed methods is evaluated in multiple experiments performed with a physical robot in the context of handle grasping, object grasping, and object pulling. The experimental results suggest that execution models contribute towards avoiding execution failures, but also represent a first step towards more introspective robots that are able to analyse some of their execution failures in an explicit manner.

Explicit-to-Implicit Robot Imitation Learning by Exploring Visual Content Change

Demonstration understanding is the vital component for robot imitation learning. In this work, we investigate the visual change-based representation of the demonstration and build imitation learning pipelines in both explicit and implicit ways. Specifically, we first propose to represent the demonstration video via the visual change map and utilize it to generate explicit commands for robot execution. To pursue a more <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">“humanlike”</i> imitation learning pipeline, an implicit method is presented by extending the visual change-based representation from image level to feature level. Extensive experiments are conducted to evaluate the proposed methods and the results show that the proposed visual change-based approaches achieve the state-of-the-art imitation learning performance. Also, the results indicate the superiority of the implicit method over the explicit one for imitation learning. Supplementary video is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://vsislab.github.io/explicit2implicit/</uri> .

Safety-Aware Time-Optimal Motion Planning With Uncertain Human State Estimation

Human awareness in robot motion planning is crucial for seamless interaction with humans. Many existing techniques slow down, stop, or change the robot's trajectory locally to avoid collisions with humans. Although using the information on the human's state in the path planning phase could reduce future interference with the human's movements and make safety stops less frequent, such an approach is less widespread. This paper proposes a novel approach to embedding a human model in the robot's path planner. The method explicitly addresses the problem of minimizing the path execution time, including slowdowns and stops owed to the proximity of humans. For this purpose, it converts safety speed limits into configuration-space cost functions that drive the path's optimization. The costmap can be updated based on the observed or predicted state of the human. The method can handle deterministic and probabilistic representations of the human state and is independent of the prediction algorithm. Numerical and experimental results on an industrial collaborative cell demonstrate that the proposed approach consistently reduces the robot's execution time and avoids unnecessary safety speed reductions.

Aquablation Treatment for Benign Prostate Hyperplasia: Current Standardized Procedure.

The following video Atlas summary reviews all technical elements of the standardized setup, robotic execution, aquablation procedure, and hemostasis for efficient rapid benign prostate hyperplasia treatment.

All electrospun fabrics based piezoelectric tactile sensor

Tactile sensors have been widely used in the areas of health monitoring and intelligent human-machine interface. Flexible tactile sensors based on nanofiber mats made by electrospinning can meet the requirements of comfortability and breathability for wearing the body very well. Here, we developed a flexible and self-powered tactile sensor that was sandwich assembled by electrospun organic electrodes and a piezoelectric layer. The metal-free organic electrodes of thermal plastic polyurethane (PU) nanofibers decorated with multi-walled carbon nanotubes were fabricated by electrospinning followed by ultrasonication treatment. The electrospun polyvinylidene fluoride-trifluoroethylene (PVDF-TrFE) mat was utilized as the piezoelectric layer, and it was found that the piezoelectric performance of PVDF-TrFE nanofiber mat added with barium titanate (BaTiO3) nanoparticles was enhanced about 187% than that of the pure PVDF-TrFE nanofiber mat. For practical application, the as-prepared piezoelectric tactile sensor exhibited an approximative linear relationship between the external force and the electrical output. Then the array of fabricated sensors was attached to the fingertips of a glove to grab a cup of water for tactile sensing, and the mass of water can be directly estimated according to the outputs of the sensor array. Attributed to the integrated merits of good flexibility, enhanced piezoelectric performance, light weight, and efficient gas permeability, the developed tactile sensor could be widely used as wearable devices for robot execution end or prosthesis for tactile feedback.

Energy-Efficient Robot Configuration and Motion Planning Using Genetic Algorithm and Particle Swarm Optimization

The implementation of Industry 5.0 necessitates a decrease in the energy consumption of industrial robots. This research investigates energy optimization for optimal motion planning for a dual-arm industrial robot. The objective function for the energy minimization problem is stated based on the execution time and total energy consumption of the robot arm configurations in its workspace for pick-and-place operation. Firstly, the PID controller is being used to achieve the optimal parameters. The parameters of PID are then fine-tuned using metaheuristic algorithms such as Genetic Algorithms and Particle Swarm Optimization methods to create a more precise robot motion trajectory, resulting in an energy-efficient robot configuration. The results for different robot configurations were compared with both motion planning algorithms, which shows better compatibility in terms of both execution time and energy efficiency. The feasibility of the algorithms is demonstrated by conducting experiments on a dual-arm robot, named as duAro. In terms of energy efficiency, the results show that dual-arm motions can save more energy than single-arm motions for an industrial robot. Furthermore, combining the robot configuration problem with metaheuristic approaches saves energy consumption and robot execution time when compared to motion planning with PID controllers alone.

Highly-automated, high-throughput replication of yeast-based logic circuit design assessments.

We describe an experimental campaign that replicated the performance assessment of logic gates engineered into cells of Saccharomyces cerevisiae by Gander et al. Our experimental campaign used a novel high-throughput experimentation framework developed under Defense Advanced Research Projects Agency’s Synergistic Discovery and Design program: a remote robotic lab at Strateos executed a parameterized experimental protocol. Using this protocol and robotic execution, we generated two orders of magnitude more flow cytometry data than the original experiments. We discuss our results, which largely, but not completely, agree with the original report and make some remarks about lessons learned. Graphical