Real-time Robotic System Research Articles

Modified Bilateral Active Estimation Model: A Learning-Based Solution to the Time Delay Problem in Robotic Tele-Control

The ubiquitous presence of three types of delay in robotic teleoperation systems, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , computation delay, transmission delay, and mechanical delay, is the major factor of the system degradation. It is noticeable that the transmission latency over the communication network shows a periodic trend due to the network flux changing. Accordingly, in our previous work, a neural network-based open-loop approach named Bilateral Active Estimation Model (BAEM) was proposed to compensate for the upcoming transmission delay in a unilateral teleoperation system by sending predicted trajectories as commands. In this paper, a modified version of BAEM (m-BAEM) is proposed to compensate for all these three types of delay explicitly, and a real-time robotic teleoperation system based on Robot Operating System 2 (ROS 2) framework is built to evaluate the performance of the m-BAEM in constant and varying delay scenarios with both pre-defined and human-input trajectories. The results of pre-defined trajectories present the satisfactory performance of the m-BAEM even in the presence of transmission delay up to 1000 miliseconds with large variations. The main limitation of the m-BAEM is that it is yet unable to handle unknown trajectories.

Real-time robotic weed knife control system for tomato and lettuce based on geometric appearance of plant labels

Automated weed management tools in vegetable crops are needed to reduce or eliminate hand-weeding because of labour shortages and cost. Distinguishing crop plants from weeds in complex natural scenes of crop-weed mixtures remains a challenge for weed management automation. This paper presents a novel solution to the weed control problem by employing crop signalling technology: a novel systems approach that creates a machine-readable crop plant. A robot-vision-based weed-knife control system with a novel three-dimensional geometric detection algorithm was developed to automate weed control for tomato and lettuce crops. The system successfully detected the crop signal from occluded crop plants while traveling at speeds up to of 3.2 km h−1. The in-field experiments show that the system is able to reduce the number of weed plants by 83% in the seedling area. Crop detection accuracy was measured at 97.8% (precision 0.998 and recall 0.952) with a detection time of 30 ms f−1. This paper also shows that the crop signalling system has the advantage that prior knowledge of visual features of each crop and weed species is not required and poor visual appearance of the crop plants or weeds does not affect system performance.

Single image dehazing based on multiscale product prior and application to vision control

In this paper, a novel dehazing algorithm based on multiscale product (MSP) prior is presented. First, the observed hazy image is decomposed into its approximation and detail subbands by undecimated Laplacian decomposition. Then the MSPs of the approximation subbands for each band of the image are calculated to obtain the MSP prior. This prior keeps the significant information of the image, whereas it is capable of detecting the haze in the image. By the use of this prior and the hazy image model, a fast and robust dehazing algorithm is obtained. The proposed method is compared with commonly used methods. The results demonstrate that the proposed algorithm is better than the former methods. Being a fast and robust algorithm, the proposed method has also been applied to a real-time robotic vision control system.

RAPID PROTOTYPING OF A GAIT GENERATION METHOD USING REAL-TIME HARDWARE IN LOOP SIMULATION

Formal control design techniques usually rely on the plant model of a desired system. The plant model can be derived from the foundation principles of the system but often includes unidentified parameters. In order to approximate these unknown parameters, experiments are conducted to collect information from the behavior of plant dynamics. Thus, closed-loop feedback control methods can be formulated upon the estimation of an appropriate plant model using simulation techniques. Rapid prototyping techniques support this design paradigm which requires that the rapid prototype operate in real time, interact with real hardware and have supporting control functionality. This paper delivers a study on the modeling and testing of a gait generation method using a rapid prototyping technique. The objective is to develop a method of rapid prototyping to test new walking algorithms on a real-time robotic system. The paper addresses the drawbacks of a non-real-time simulation by evaluating the locomotion of a six-legged robot in terms of tracking errors and signifies the proposed methodology of tuning the gait generation algorithm online through real-time hardware in loop simulation setup. Finally, the paper inspects the improved locomotion of the robot using the proposed methodology and signifies it as a valid prototyping technique for approximating novel gait generation algorithms on real-time robotic systems.

Single-session lung radiosurgery using robotic image-guided real-time respiratory tumor tracking.

Objective: To describe the clinical feasibility for single session lung radiosurgery using a real-time robotic system with respiratory tracking. Methods: We treated 62 patients with 75 lung tumors with robotic radiosurgery (CyberKnife®, Accuray). Twenty-two patients had primary tumors and 40 had metastatic tumors. Forty-two patients underwent computed tomography-guided percutaneous placement of one fiducial directly into the tumor. Ten patients were treated using spinal tracking, and in another ten patients fiducial free lung tumor tracking (Xsight lung, Accuray Inc.) was performed. All patients were treated with single session radiosurgery to a dose of 22 - 24 Gy. The median tumor volume was 15 cc (1.5 – 64 cc). The last 23 patients were planned with a new Monte Carlo treatment planning algorithm. Follow-up CT scanning was performed every two months. Results: All patients could be treated with the automated robotic technique. The 6-, 12-, 24 actuarial local tumor control rate was 98%, 88% and 88%, respectively. The median follow up time was 10 months (range: 0.4 – 30 months). Six patients (10%) required a chest tube insertion after fiducial implantation because of pneumothorax. Four patients experienced nausea after treatment. No other short-term adverse reactions were found. Fifteen patients showed Grade 1 and one patient Grade 2 signs of pneumonitis. No Grade 3 or grade 4 toxicity was detected. Conclusions: Single-session radiosurgery for lung tumor tracking using the described technology is a stable, safe, and feasible concept for respiratory tracking of tumors during robotic lung radiosurgery in selected patients. The single fraction approach is convenient for the patient and does not interfere with concomitant systemic treatments. Longer follow-up is needed for definitive clinical results.

A Real-Time Specific Weed Recognition System Using Statistical Methods

The identification and classification of weeds are of major technical and economical importance in the agricultural industry. To automate these activities, like in shape, color and texture, weed control system is feasible. The goal of this paper is to build a real-time, machine vision weed control system that can detect weed locations. In order to accomplish this objective, a real-time robotic system is developed to identify and locate outdoor plants using machine vision technology and pattern recognition. The algorithm is developed to classify images into broad and narrow class for real-time selective herbicide application. The developed algorithm has been tested on weeds at various locations, which have shown that the algorithm to be very effectiveness in weed identification. Further the results show a very reliable performance on weeds under varying field conditions. The analysis of the results shows over 90 percent classification accuracy over 140 sample images (broad and narrow) with 70 samples from each category of weeds.

Technical Description, Phantom Accuracy, and Clinical Feasibility for Single-Session Lung Radiosurgery Using Robotic Image-Guided Real-time Respiratory Tumor Tracking

To describe the technological background, the accuracy, and clinical feasibility for single session lung radiosurgery using a real-time robotic system with respiratory tracking. The latest version of image-guided real-time respiratory tracking software (Synchrony, Accuray Incorporated, Sunnyvale, CA) was applied and is described. Accuracy measurements were performed using a newly designed moving phantom model. We treated 15 patients with 19 lung tumors with robotic radiosurgery (CyberKnife, Accuray) using the same treatment parameters for all patients. Ten patients had primary tumors and five had metastatic tumors. All patients underwent computed tomography-guided percutaneous placement of one fiducial directly into the tumor, and were all treated with single session radiosurgery to a dose of 24 Gy. Follow up CT scanning was performed every two months. All patients could be treated with the automated robotic technique. The respiratory tracking error was less than 1 mm and the overall shape of the dose profile was not affected by target motion and/or phase shift between fiducial and optical marker motion. Two patients required a chest tube insertion after fiducial implantation because of pneumothorax. One patient experienced nausea after treatment. No other short-term adverse reactions were found. One patient showed imaging signs of pneumonitis without a clinical correlation. Single-session radiosurgery for lung tumor tracking using the described technology is a stable, safe, and feasible concept for respiratory tracking of tumors during robotic lung radiosurgery in selected patients. Longer follow-up is needed for definitive clinical results.

Real-time haptic-teleoperated robotic system for motor control analysis

A versatile teleoperated robotic system was created as an assessment device for testing upper-extremity motor control adaptation using different control strategies. While many systems display output virtually on a computer monitor, this system was designed to output in three-dimensional physical space. The system accepts haptic force and torque input, and outputs robot end-effector displacements and rotations in three spatial dimensions. Benefits of this system include flexibility to conduct a variety of dissimilar tasks and reality of user feedback in physical space. Two separate experiments validated the teleoperated robotic system. The first experiment tested unimanual human motor control and the second tested bimanual motor control. This teleoperated robotic system can be used as an assessment device to study neuromuscular adaptability via a variety of control strategies providing a new and functional approach to human motor control analysis.

Grounded spoken language acquisition: experiments in word learning

Language is grounded in sensory-motor experience. Grounding connects concepts to the physical world enabling humans to acquire and use words and sentences in context. Currently most machines which process language are not grounded. Instead, semantic representations are abstract, pre-specified, and have meaning only when interpreted by humans. We are interested in developing computational systems which represent words, utterances, and underlying concepts in terms of sensory-motor experiences leading to richer levels of machine understanding. A key element of this work is the development of effective architectures for processing multisensory data. Inspired by theories of infant cognition, we present a computational model which learns words from untranscribed acoustic and video input. Channels of input derived from different sensors are integrated in an information-theoretic framework. Acquired words are represented in terms of associations between acoustic and visual sensory experience. The model has been implemented in a real-time robotic system which performs interactive language learning and understanding. Successful learning has also been demonstrated using infant-directed speech and images.