Remote Robot Operation Research Articles

Innovative Application of Optoelectronic Control Engineering for Safety Assurance of Surgical Robots

As an emerging technology, optoelectronic control engineering has demonstrated a wide range of application potential in surgical robot safety and security. Through a literature review, this paper provides an in-depth discussion on the innovative applications of optoelectronics technology in enhancing the positioning accuracy, environment sensing, supple control and remote operation of surgical robots. At the same time, this paper summarizes the current research hotspots and future research directions, with a view to providing a reference for promoting the technological progress in this field. The current research focuses on improving the operating precision of surgical robots, enhancing the environment sensing ability and the application of force feedback system, while the future research will mainly focus on interdisciplinary cooperation and new technology development.

VR map construction for orchard robot teleoperation based on dual-source positioning and sparse point cloud segmentation

The unstructured nature of orchard environments presents significant challenges for autonomous navigation of orchard robots. Teleoperation, combined with virtual reality (VR), has emerged as a promising solution to overcome the limitations of on-board autonomous navigation capabilities in general-purpose robots. However, constructing accurate and semantically meaningful VR maps for orchard environments remains a challenge. This study proposes a novel VR map construction framework for orchard robot teleoperation visualization. It comprises two key components: dual-source combined positioning and semantic segmentation based on sparse point clouds. First, a novel sliding window-based data fusion approach is proposed to address the positioning challenges in semi-obstructed orchard environments. This approach combines GNSS data and laser matching to achieve robust and accurate positioning for orchard robots. Second, the framework capitalizes on the installation characteristics of mobile robot radars to achieve real-time classification of sparse point clouds. This allows for the removal of unstable elements, such as leaves, which would hinder effective teleoperation. By integrating these components, a VR map suitable for VR teleoperation of orchard robots is achieved. To validate the effectiveness of the proposed method, a VR teleoperation platform was constructed. The experimental results demonstrate that the method successfully fulfills the fundamental requirements for map visualization during VR remote operation of orchard robots. Furthermore, this study provides a valuable reference for the application of digital twins in agricultural robotics.

Design of a robotic system to assist in the treatment of severe COVID‐19 patients

AbstractThe article presents the conceptualization, development, and implementation of a sophisticated mobile robotic system with the purpose of providing aid in the medical care of those afflicted with severe cases of COVID‐19. The robotic system will engage in verbal communication with the patient and provide updates regarding the external environment. Additionally, it will facilitate the delivery of food, beverages, and other consumable items to the isolation room. The robot has the capability to navigate to its intended location using two distinct modes: autonomous mode and online control mode. The hardware is constructed within a mobile robot system that is interconnected to the Internet over a 4G mobile network. The system employs a client–server software architecture wherein the transmission of data between the client and the server occurs through various transport protocols. Experimental simulations were conducted in an active treatment room to evaluate the performance of autonomous operating mechanisms, including obstacle avoidance positioning, safe destination navigation, and feedback display for remote robot operation.

The Impact of Virtual Reality Integration in Robotics: Enhancing Efficiency, Safety, and Human-Robot Interaction

In this paper, a review of the implementation of Virtual Reality (VR) with robotics and how it has changed various areas is explored. In general, VR improves efficiency and safety in operational work—especially in the energy sector, where it optimizes substation operations (e.g., increasing accuracy by up to 25% due to a higher level of automation). For mechatronics, VR allows advanced digital design, such as virtual prototyping and collaborative simulation, which significantly increase system efficiency. In education, VR technology has increased student engagement and learning outcomes by up to 40%, improving participation in subjects such as natural sciences. VR integration into robotics also enables the testing of various movements and trajectories for safety, resulting in minimal damages incurred during the design phase and increased productivity. VR also enhances human-robot interaction by enabling intuitive control based on better gesture recognition and a 35% increase in precision. In future applications, the remote operation of robots in dangerous environments will greatly improve emergency rescue efficiency and speed. Using VR for substation operations has translated to 25% better accuracy and a similar gain in terms of operational efficiency. By providing a more connected, efficient future in multiple spheres and delivering about 40% fewer accidents during simulations, the fusion of VR with robotics is an overall successful venture.

Models of Quality of Service Indicators for Traffic (Robots-Manipulators)

This article discusses the remote operation of robots under the control of an operator using feedback. When transferring data over the network, the quality of tasks can be degraded due to delays and packet loss. Which leads to incorrect operation of the robot and its mechanisms. To solve these problems, it is necessary to control Quality of Service and Quality of Experience. First, the following problems were onsidered: the relationship between Quality of Servic and Quality of Experience, the differences between a person and a robot, the presence of the Quality of Exerience of the robot. Further, the parameters for evaluating the quality of service for robots are defined, and what key performance indicators are needed for the quality of service model for robo s. The requirements for the quality of service for traffic de-pending on the class of industrial robots are presented. In conclusion, an assessment of the requirements for indicators for the transmission of commands from the user to the manipulator with feedback was considered.

Reappraisal of telesurgery in the era of high-speed, high-bandwidth, secure communications: Evaluation of surgical performance in local and remote environments.

Communication and video transmission delays negatively affect telerobotic surgery. Since latency varies by communication environment and robot, to realize remote surgery, both must perform well. This study aims to examine the feasibility of telerobotic surgery by validating the communication environment and local/remote robot operation, using secure commercial lines and newly developed robots. Hirosaki University and Mutsu General Hospital, 150 km apart, were connected via a Medicaroid surgical robot. Ten surgeons performed a simple task remotely using information encoding and decoding. The required bandwidth, delay time, task completion time, number of errors, and image quality were evaluated. Next, 11 surgeons performed a complex task using gallbladder and intestinal models in local/remote environments; round trip time (RTT), packet loss, time to completion, operator fatigue, operability, and image were observed locally and remotely. Image quality was not so degraded as to affect remote robot operation. Median RTT was 4msec (2-12), and added delay was 29 msec. There was no significant difference in accuracy or number of errors for cholecystectomy, intestinal suturing, completion time, surgeon fatigue, or image evaluation. The fact that remote surgery succeeded equally to local surgery showed that this system has the necessary elemental technology for widespread social implementation.

Problems to be solved in the development from the perspective of future application scenarios of telesurgery

Telemedicine, which integrates medicine, communication, engineering, information and other disciplines, is a hot emerging cross field in recent years. With the development of telecommunication technology and surgical robot, telesurgery is regarded as the "crown pearl" in telemedicine and has attracted more and more attention. As an extension of traditional surgery, telesurgery greatly extends the connotation and concept of surgery and embodies the great leap forward development of surgical technology. Despite the current limitations such as network delay, transparency of remote robot operation and team construction of surgeons, telesurgery has still formed a variety of innovative application scenarios and achieved rapid development in China in recent years. In view of the uneven distribution of medical resources in China and the epidemic of COVID-19 in the world, this paper puts forward the possible problems and solutions in the development of telesurgery, and looks forward to the feasibility of telesurgery technology in process of shifting the focus of medical and health care down to the community level, channeling resources accordingly.

Augmented reality-based autostereoscopic surgical visualization system for telesurgery.

The visualization of remote surgical scenes is the key to realizing the remote operation of surgical robots. However, current non-endoscopic surgical robot systems lack an effective visualization tool to offer sufficient surgical scene information and depth perception. We propose a novel autostereoscopic surgical visualization system integrating 3D intraoperative scene reconstruction, autostereoscopic 3D display, and augmented reality-based image fusion. The preoperative organ structure and the intraoperative surface point cloud are obtained from medical imaging and the RGB-D camera, respectively, and aligned by an automatic marker-free intraoperative registration algorithm. After registration, preoperative meshes with precalculated illumination and intraoperative textured point cloud are blended in real time. Finally, the fused image is shown on a 3D autostereoscopic display device to achieve depth perception. A prototype of the autostereoscopic surgical visualization system was built. The system had a horizontal image resolution of 1.31mm, a vertical image resolution of 0.82mm, an average rendering rate of 33.1 FPS, an average registration rate of 20.5 FPS, and average registration errors of approximately 3mm. A telesurgical robot prototype based on 3D autostereoscopic display was built. The quantitative evaluation experiments showed that our system achieved similar operational accuracy (1.79 ± 0.87mm) as the conventional system (1.95 ± 0.71mm), while having advantages in terms of completion time (with 34.11% reduction) and path length (with 35.87% reduction). Post-experimental questionnaires indicated that the system was user-friendly for novices and experts. We propose a 3D surgical visualization system with augmented instruction and depth perception for telesurgery. The qualitative and quantitative evaluation results illustrate the accuracy and efficiency of the proposed system. Therefore, it shows great prospects in robotic surgery and telesurgery.

The influence of gaming frequency and viewing perspective on a remote robot operation task

The influence of gaming frequency and viewing perspective on a remote robot operation task

Multi-hop wireless command and telemetry communication system for remote operation of robots with extending operation area beyond line-of-sight using 920 MHz/169 MHz

The role of robot technologies is growing particularly in the areas of disaster response, infrastructure inspection, and so on. For remote operation of the robots, wireless communication is important to control, send command, and receive telemetry signals. However, it could be often disconnected due to radio wave attenuation, shadowing, or interference in the real field. In order to overcome the above problems, we have proposed and developed a multi-hop communication technology, specialized for remotely controlled robots, including drones, with high reliability and low latency using dual frequency band in 920 and 169 MHz. In this paper, we evaluate the radio propagation and communication quality characteristics in field experiment. In the experiment, the application of the proposed technology to disaster response was demonstrated.

A Survey of End-to-End Solutions for Reliable Low-Latency Communications in 5G Networks

Very low latency and high reliability are two of the main requirements for new applications exploiting 5G networks. This is the case for the remote operation of robots or vehicles, the autonomous interaction of equipment in a factory, autonomous driving and tactile internet applications. Although the TCP/IP stack has been sufficient as the end-to-end solution for most of the history of the Internet, a number of surveys have appeared recently presenting many different methods for managing the end-to-end communication to meet the requirements of various technologies such as that of 5G networks. In this paper, we present a novel classification of the literature focused on new end-to-end solutions and the creation of services towards the support of low latency (1 ms) and high reliability (10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-9</sup> error rate) in current and future 5G networks. We specifically highlight how the proposals can be classified according to enabling technologies and the specific method used to achieve success in terms of the latency and reliability. The literature related to end-to-end solutions for reliable low-latency communications are organized according to three main topics: (i) end-to-end protocols that improve communication in terms of latency and reliability, (ii) functionality or technologies implemented on the network to support the current demands, and (iii) application programming interfaces that enhance the correct utilization of those protocols and additional technologies.

Simultaneous bipedal locomotion based on haptics for teleoperation

ABSTRACT This paper proposes a novel, simultaneous bipedal locomotion method using haptics for remote operation of biped robots. In general, traditional biped walking methods require very high computational power and advanced controllers to perform the required task. However, in this proposed method, a master exoskeleton attached to the human’s lower body is used to obtain the trajectory and haptic information to generate the trajectory of the slave biped robot in real time. Lateral motion of the center of mass of the biped is constrained in this experiment. Also, it is considered that no communication delay is presented in between the two systems in this experiment, and they are not discussed in this paper. Since a direct motion transmission is used in the proposed method, this method is quite straight forward and a simultaneous walking can be realized at the same time with high performance. Also, it does not require an exact dynamic model of the biped or specific method to plan the trajectory. The gait pattern of the biped is directly determined by that of the human. Also, the operator can feel the remote environment through the exoskeleton robot. Results obtained from the experiments validate the proposed method.

On the design and implementation of a dual fisheye camera-based surveillance vision system

Image processing and computer vision have been a focus of researchers for decades in various application domains. This research is continuously rising with the rise of Artificial Intelligence in the fourth industrial revolution. One of the important digital image processing applications is to produce panorama images. The wide range of view a panorama image provides can be used in a variety range of applications which may include surveillance applications and remote robot operations. A panorama image is a combination of several individual natural looking images into a composite one to provide a wide field of view that may reach 360 degrees horizontally without any distortion. Wide-angle lenses provide a wide field of view, but using them alone does not necessarily make a panorama image. In this work the design and implementation of a wide-angle stereo vision system that suites many real-time applications is proposed. The system makes use of two wide-angle fisheye cameras where each camera covers around 170 degrees field of view. The horizontal angle between the cameras is 140 degrees. The cameras acquire the instantaneous overlapping images continuously and transmits them to a base station via a communication link. The base station calibrates, corrects, correlates and stitches the non-overlapping corrected images to a composite one. The resultant final image covers 310 degrees field of view. The system is of low computational complexity compared with previously implemented systems. It is tested on a laptop and on a standalone embedded computing device. The processing speed for the panorama image stitching including the correction of the fisheye barrel distortion on the laptop computer and the embedded computer is 11 fps, and 6 fps, respectively.

An Evaluation Model Based on the Fuzzy AHP for Teleoperation Performance

ABSTRACTObjective: This study is to construct a model composed of multiple indices to measure astronaut teleoperation performance.Background: Teleoperation is a key operation during spaceflight. Comprehensive scientific evaluation techniques for teleoperation performance are important for astronaut selection and training.Method: Four categories of indices, i.e., completion performance, safety performance, subjective cognitive performance and physiological parameters, were adopted in this model. The fuzzy analytic hierarchy process (AHP) was used to compute the weight coefficients of the indices to construct the linear performance model. Sixty-four male subjects were recruited to complete three teleoperation tasks, including point aiming, line alignment, and obstacle avoidance, in simulated environments. Their performance was rated by the researchers after the tasks.Results: The safety performance index had the highest weight coefficient (0.5274) of the first level indices, and the weight coefficient of the collisions index (0.8227) was the highest of the second level indices for safety performance. The linear performance model was constructed to calculate comprehensive performance values.Conclusion: Safety considerations are the most important factors for astronauts to consider in teleoperation. The collision of the robotic arm is the most vital safety index. The comprehensive value derived from the model was significantly related to overall performance. The validity of our model was verified, and it is anticipated that the model will be used to evaluate the performance of remote robotic arm operations.

Dependable wireless connectivity: insights and methods for 5G and beyond

Dependability is a measure of availability and reliability of systems/services. In the context of communication systems, dependability is governed by the coverage probability of the network under prescribed service requirements, by the latency of data transmissions as well as by the transmission error probability. Achieving dependable connectivity can be very challenging, especially within wireless mobile communications, where the transmission channel is often prone to severe fading and strong interference. Current generations of cellular mobile communication systems (4G and below) can mainly provide best effort services and are not well equipped to achieve a sufficiently high level of dependability as required by many novel applications, such as, road-safety relevant information exchange in vehicular communications, as well as, wireless remote operation of robots and drones. Standardization bodies have already recognized the market potential of such use-cases for mobile communications, and correspondingly efforts are ongoing to enhance the fifth generation of cellular systems (5G) towards ultra-reliable low-latency transmission. In this paper, we provide insights gained by our research work within the Christian Doppler Laboratory for Dependable Wireless Connectivity for the Society in Motion with respect to factors influencing the dependability of 5G mobile cellular systems, and we present our achievements over the past two years for enhancing the robustness, reliability and efficiency of dependable wireless communications.

The development of remote robot operation method of hand sign

The development of remote robot operation method of hand sign

Corrigendum

Vasey, E., FakhrHosseini, M. S., Zheng, Z., Park, C.-H., Howard, A., Jeon, M. Development and Usability Testing of a Remote Control App for An Interactive Robot. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 61, 1: pp. 808-812. First Published September 28, 2017. (Original DOI: 10.1177/1541931213601695) Please note that this article published with errors. The differences are in the first paragraph in the Related Works section. In the published version, it reads: The remote operation of robots using mobile applications has been successfully implemented in other experiments. Trigo and Brown [10] developed an app for Android and iOS to help teachers remotely control a Nao robot to assist in teaching children with intellectual disabilities.This app relied on an external server using the Python programming language and could make Nao change its posture and gesture, walk, ask pre-recorded questions, as well as execute several pre-stored dance routines. An evaluation was done using a think-aloud protocol with several teachers, and overall feedback was positive. However, this app neither allowed for creation of new animations within the app, nor did it allow for teachers to record their own sets of phrases, questions, or motions. The new version reads (differences in bold): The remote operation of robots using mobile applications has been successfully implemented in other experiments. Trigo and Brown [10] developed an app for Android and iOS to help teachers remotely control a Nao robot to assist in teaching children with intellectual disabilities. This app could make Nao change its posture and gesture, walk, ask user-defined questions as well as execute several pre-stored dance routines. An evaluation was done using a think-aloud protocol and a questionnaire with several teachers, and overall feedback was positive. However, this app did not allow for creation of new animations within the app or for teachers to record their own motions (M. Trigo, personal communication, October 31, 2017).

移動ロボットの遠隔操縦インタフェースのための全方位映像と三次元形状モデルを用いた情報提示手法(「バーチャルVR～VR技術の応用・転用・新展開」特集)

移動ロボットの遠隔操縦インタフェースのための全方位映像と三次元形状モデルを用いた情報提示手法(「バーチャルVR～VR技術の応用・転用・新展開」特集)

Intuitive Terrain Reconstruction Using Height Observation-Based Ground Segmentation and 3D Object Boundary Estimation

Mobile robot operators must make rapid decisions based on information about the robot’s surrounding environment. This means that terrain modeling and photorealistic visualization are required for the remote operation of mobile robots. We have produced a voxel map and textured mesh from the 2D and 3D datasets collected by a robot’s array of sensors, but some upper parts of objects are beyond the sensors’ measurements and these parts are missing in the terrain reconstruction result. This result is an incomplete terrain model. To solve this problem, we present a new ground segmentation method to detect non-ground data in the reconstructed voxel map. Our method uses height histograms to estimate the ground height range, and a Gibbs-Markov random field model to refine the segmentation results. To reconstruct a complete terrain model of the 3D environment, we develop a 3D boundary estimation method for non-ground objects. We apply a boundary detection technique to the 2D image, before estimating and refining the actual height values of the non-ground vertices in the reconstructed textured mesh. Our proposed methods were tested in an outdoor environment in which trees and buildings were not completely sensed. Our results show that the time required for ground segmentation is faster than that for data sensing, which is necessary for a real-time approach. In addition, those parts of objects that were not sensed are accurately recovered to retrieve their real-world appearances.

Teleoperation of Multiple Social Robots

Teleoperation of multiple robots by a single operator has been studied extensively for applications such as search and navigation; however, this concept has never been applied to the field of social, conversational robots. In this paper, we explore the unique challenges posed by the remote operation of multiple social robots, where an operator must perform auditory multitasking to assist multiple interactions at once. It describes the general system requirements in four areas: social human-robot interaction (HRI) design, autonomy design, multirobot coordination, and teleoperation interface design. Based on this design framework, we have developed a system in which a single operator can simultaneously control four robots in conversational interactions with users. Key elements of our implementation include a control architecture enabling the scripting of conditional behavior flows for social interaction, a graphical interface enabling an operator to control one robot at a time while monitoring several others in the background, and a technique called “proactive timing control,” which is an automated method for smoothly interleaving the demands of multiple robots for the operator's attention. We also present metrics for describing and predicting robot performance, and we show experimental results demonstrating the effectiveness of our system through simulations and a laboratory experiment based on real-world interactions.