Vehicle Teleoperation Research Articles

Performance of Successive Reference Pose Tracking vs Smith Predictor Approach for Direct Vehicle Teleoperation Under Variable Network Delays

Performance of Successive Reference Pose Tracking vs Smith Predictor Approach for Direct Vehicle Teleoperation Under Variable Network Delays

A mathematical analysis of teleoperator optimization using multiple sensors

As the role of remote vehicle operation gains prominence, ensuring robust wireless communication systems is crucial. Prior studies have delved into various aspects of vehicle teleoperation, including its legal framework and workforce implications. Some have even formulated models to estimate the number of remote operators for large automated vehicle fleets. These models suggest that automated vehicles overseen by remote operators could supplant a significant portion of current driving jobs in the United States. Such findings underscore the need for governmental review and policy adjustment to better accommodate the burgeoning field of teleoperation, which could substantially accelerate the advancement of vehicle automation. The present study focuses on evaluating the reliability of a bi-directional communication channel based on key variable parameters. This assessment aids in determining the optimal parameter settings, thereby enhancing the effectiveness of teleoperation systems.

Predictive Display With Perspective Projection of Surroundings in Vehicle Teleoperation to Account Time-Delays

Teleoperation provides human operator sophisticated perceptual and cognitive skills into an over the network control loop. It gives hope of addressing some challenges related to vehicular autonomy which is based on artificial intelligence by providing a backup plan. Variable network time delays in data transmission is the major problem in teleoperating a vehicle. On 4G network, variability of these delays is high. Due to this, both video streaming and driving commands encounter variable time delay. This paper presents an approach of providing the human operator a forecast video stream which replicates future perspective of vehicle field of view accounting the delay present in the network. Regarding the image transformation, perspective projection technique is combined with correction given by smith predictor in the control loop. This image transformation accounts current time delay and tries to address both issues, time delays as well as its variability. For experiment sake, only frontward field of view is forecast. Performance is evaluated by performing online vehicle teleoperation on street edge case maneuvers and later comparing the path deviation with and without perspective projection.

Structure‐Aware Image Translation‐Based Long Future Prediction for Enhancement of Ground Robotic Vehicle Teleoperation

Predicting future frames through image‐to‐image translation and using these synthetically generated frames for high‐speed ground vehicle teleoperation is a new concept to address latency and enhance operational performance. In the immediate previous work, the image quality of the predicted frames was low and a lot of scene detail was lost. To preserve the structural details of objects and improve overall image quality in the predicted frames, several novel ideas are proposed herein. A filter has been designed to remove noise from dense optical flow components resulting from frame rate inconsistencies. The Pix2Pix base network has been modified and a structure‐aware SSIM‐based perpetual loss function has been implemented. A new dataset of 20 000 training input images and 2000 test input images with a 500 ms delay between the target and input frames has been created. Without any additional video transformation steps, the proposed improved model achieved PSNR of 23.1; SSIM of 0.65; and MS‐SSIM of 0.80, a substantial improvement over our previous work. A Fleiss’ kappa score of >0.40 (0.48 for the modified network and 0.46 for the perpetual loss function) proves the reliability of the model.

Designing Gaze-Based Interactions for Teleoperation: Eye Stick and Eye Click

Perspective-taking and attentional switching are some of the ergonomic challenges that existing teleoperation human-machine interface designs need to address. This study developed two gaze interaction methods, the Eye Stick and the Eye Click, which were based on the joystick metaphor and the navigation metaphor, respectively, to be used in exocentric perspective teleoperation scenarios. We conducted two user studies to test the task performance and the subjective experience of the gaze interaction methods in a virtual ground vehicle teleoperation task. The results showed that compared with a traditional joystick design, the Eye Stick led to a shorter driving distance and the Eye Click led to less task time, and the gaze interaction methods had performance advantages in more difficult mazes. After multiple task sessions, the participants reported that the gaze interaction methods and the traditional joystick were similar in terms of task workload, perceived learnability, and satisfaction; however, the perceived usability of the Eye Stick was not as good as the Eye Click and the traditional joystick. In conclusion, both the Eye Stick and the Eye Click are feasible and promising gaze interaction methods for teleoperation applications with task performance advantages; however, more research is needed to optimize their user experience design.

Virtual Yaw Rate Display for Reducing Steering Instability of Remote Driving

In vehicle teleoperation, reducing instability of driving caused by communication delay is one of important technical problems. This study proposes a passivity-based vehicle teleoperation system that compensates for the delay of visual information presented to the driver. The proposed method applies wave variable transformation to steering angle and yaw rate, and presents modified yaw rate to the driver by altering the viewing direction of camera images. This effectively makes the driver feel that the vehicle responds to his/her steering operation without delay, and consequently over-steering is greatly reduced. The results of experiments using a driving simulator showed that the proposed method greatly improves lateral stability of driving.

A framework for usability evaluation of automated driving human-machine interface in context of UAE

As a future mobility enabler, vehicle teleoperation offers a viable option for efficiently leveraging the benefits of driverless cars, whilst fully automated cars (SAE 5) are not completely viable. Despite the huge amount of research on self-driving technology, the notion of riding in a fully automated vehicle is still sceptical. Hence necessitating the presence of a remote operator who tele monitors the vehicle and acquire control in case of failure beyond the capabilities of vehicle automation, such as misuse, disuse or mode confusion and ensuring safety and reliability. Keeping the current inclination towards automated vehicles and the launch of the United Arab Emirates” (UAE) first driverless, autonomous 12-seater shuttle, the researcher has suggested a low-fidelity basic prototype. To connect the vehicle’s automation and human teleoperation, the researcher has proposed a unique and interactive framework of Human-Machine Interface (HMI) to cater teleoperation in the context of automated vehicles (AVs) in the UAE. The design is optimised for the teleoperation of a highly automated shuttle by a public transportation control centre based on a logical and careful inquiry. After deriving a detailed analysis of requirements, a prototype is then created and refined until an interactive user-interface emerges. Fifteen control station experts are presented with a prototype for evaluation in a regular manner and solicited to work out three scenarios with disruptions in the system. The prototype is then evaluated using questionnaires and structured interviews. The results validate the suggested HMI design for the remote operation of a highly automated vehicle. Participants’ feedback shows a particularly high level of satisfaction in terms of perceived capacity, feasibility, usability approval and situational responsiveness. The findings also provide valuable information for an advanced interactive HMI design and pave the way for potential future directions in further research as well.

Vehicle Teleoperation: Human in the Loop Performance Comparison of Smith Predictor with Novel Successive Reference-Pose Tracking Approach

Vehicle teleoperation has the ability to bridge the gap between completely automated driving and manual driving by remotely monitoring and operating autonomous vehicles when their automation fails. Among many challenges related to vehicle teleoperation, the considered ones in this work are variable time delay, saturation of actuators installed in vehicle, and environmental disturbance, which together limit the teleoperation performance. State-of-the-art predictive techniques estimate vehicle states to compensate for the delays, but the predictive states do not account for sudden disturbances that the vehicle observes, which makes the human-picked steer inadequate. This inadequacy of steer deteriorates the path-tracking performance of vehicle teleoperation. In the proposed successive reference-pose-tracking (SRPT) approach, instead of transmitting steering commands, the reference trajectory, in the form of successive reference poses, is transmitted to the vehicle. This paper introduces a method of generation of successive reference poses with a joystick steering wheel and compares the human-in-loop path-tracking performance of the Smith predictor and SRPT approach. Human-in-loop experiments (with 18 different drivers) are conducted using a simulation environment that consists of the integration of a real-time 14-DOF Simulink vehicle model and Unity game engine in the presence of bidirectional variable delays. Scenarios for performance comparison are low adhesion ground, strong lateral wind, tight corners, and sudden obstacle avoidance. Result shows significant improvement in reference tracking and in reducing human effort in all scenarios using the SRPT approach.

High Latency Unmanned Ground Vehicle Teleoperation Enhancement by Presentation of Estimated Future through Video Transformation

Long-distance, high latency teleoperation tasks are difficult, highly stressful for teleoperators, and prone to over-corrections, which can lead to loss of control. At higher latencies, or when teleoperating at higher vehicle speed, the situation becomes progressively worse. To explore potential solutions, this research work investigates two 2D visual feedback-based assistive interfaces (sliding-only and sliding-and-zooming windows) that apply simple but effective video transformations to enhance teleoperation. A teleoperation simulator that can replicate teleoperation scenarios affected by high and adjustable latency has been developed to explore the effectiveness of the proposed assistive interfaces. Three image comparison metrics have been used to fine-tune and optimise the proposed interfaces. An operator survey was conducted to evaluate and compare performance with and without the assistance. The survey has shown that a 900ms latency increases task completion time by up to 205% for an on-road and 147% for an off-road driving track. Further, the overcorrection-induced oscillations increase by up to 718% with this level of latency. The survey has shown the sliding-only video transformation reduces the task completion time by up to 25.53%, and the sliding-and-zooming transformation reduces the task completion time by up to 21.82%. The sliding-only interface reduces the oscillation count by up to 66.28%, and the sliding-and-zooming interface reduces it by up to 75.58%. The qualitative feedback from the participants also shows that both types of assistive interfaces offer better visual situational awareness, comfort, and controllability, and significantly reduce the impact of latency and intermittency on the teleoperation task.

Toward Automated Vehicle Teleoperation: Vision, Opportunities, and Challenges

Recently, vehicle teleoperation (remote driving) has been gaining momentum in the industry and is playing an important role in helping self-driving cars navigate through tricky situations they cannot handle on their own. Today, vehicle teleoperation relies fully on human teleoperators. Future teleoperation systems, powered by artificial intelligence and advanced networks, such as 5G, can potentially allow a subset of the automated driving intelligence to be off-loaded from the vehicle into the cloud. Over time, the growing level of remote driving automation can enable new automated driving solutions that are not feasible today. This article presents a vision of such intelligent teleoperation and discusses its benefits and challenges.

Mathematical Model for UGV Transparency Using Laser Range Finder

ABSTRACTFor extending human capabilities to perform tasks remotely, unmanned vehicles and robots are used,this scenario is called Teleoperation. In teleoperation, human executes tasks in a remote environment. There are many applications of vehicle teleoperation in hazardous environment, military, underwater, and space exploration. Operating a vehicle or a system remotely requires supplyingthe operator in real-time with an accurate data about the operation environment. This operation is subjected to data transmission time-delay, sensors fusion and stability problems.This paper proposes a mathematical model for the time-delay compensation algorithm to achieve transparency in teleoperation using Laser scanner data. The proposed model for compensation depends on predicting the vehicle position in the future time and the environment of operation is available for the operator with respect to this predicted position. Using this algorithm in compensating time-delay in data transmission achieving transparency. Simulation and real experiments are shown to validate the proposed model.

Robotics research in Chile

The development of research in robotics in a developing country is a challenging task. Factors such as low research funds, low trust from local companies and the government, and a small number of qualified researchers hinder the development of strong, local research groups. In this article, and as a case of study, we present our research group in robotics at the Advanced Mining Technology Center of the Universidad de Chile, and the way in which we have addressed these challenges. In 2008, we decided to focus our research efforts in mining, which is the main industry in Chile. We observed that this industry has needs in terms of safety, productivity, operational continuity, and environmental care. All these needs could be addressed with robotics and automation technology. In a first stage, we concentrate ourselves in building capabilities in field robotics, starting with the automation of a commercial vehicle. An important outcome of this project was the earn of the local mining industry confidence. Then, in a second stage started in 2012, we began working with the local mining industry in technological projects. In this article, we describe three of the technological projects that we have developed with industry support: (i) an autonomous vehicle for mining environments without global positioning system coverage; (ii) the inspection of the irrigation flow in heap leach piles using unmanned aerial vehicles and thermal cameras; and (iii) an enhanced vision system for vehicle teleoperation in adverse climatic conditions.

Comparative Study of Haptic Interfaces for Bilateral Teleoperation of VTOL Aerial Robots

Force feedback is a powerful sensory cue that can provide a user, remotely teleoperating a vertical takeoff and landing (VTOL) aerial robotic vehicle, with an enhanced perception of the robot motion and interaction with the local environment. The accepted framework for mobile vehicle teleoperation, inspired by decades of research into manipulator teleoperation, maps joystick displacements to vehicle velocity set points and reflects interactive forces from the vehicle to the human operator. This paper proposes a novel admittance configured haptic interface where the force input from the user is used as a velocity set point and the joystick position is servo-controlled to reflect the vehicle velocity. This paper provides a comparative study, both analytic and experimental, of the two interfaces to investigate the performance and perceptual differences in the context of teleoperation of a VTOL aerial robot.

Gaze direction based vehicle teleoperation method with omnidirectional image stabilization and automatic body rotation control

In robot teleoperation, a robot works as a physical agent at a remote site for a robot operator. There are mainly two tasks in robot teleoperation using camera images: environment recognition using visual information and robot control according to the recognition. In this paper, we propose a gaze direction based vehicle teleoperation method with an omnidirectional image stabilization and an automatic body rotation control. In the proposed method, we manage above two tasks in the same manner that are usually treated separately. This method is an intuitive vehicle teleoperation method where an operator do not need to have concern about vehicle body orientations and can absorb differences of vehicle driving mechanisms. That is, this method frees an operator from being bothered from controlling a vehicle and the operator can concentrate on where he/she intends to go. This method mainly consists of two technologies: an omnidirectional image stabilization technology and automatic body rotation control. The conducted experiments show effectiveness of the proposed method.

Vehicle teleoperation using 3D maps and GPS time synchronization.

In conventional vehicle teleoperation systems, using low-bandwidth, high-delay transmission links causes a serious problem for remote control of the vehicles. To solve this problem, a proposed teleoperation system employs 3D maps and GPS time synchronization. Two GPS receivers measure the transmission delay, which the system uses to estimate the vehicle's location and orientation. Field experiments show that the 3D-map-based interface lets users easily comprehend the remote environment while navigating a vehicle. The experiments also show that taking communication delays into account improves maneuverability.

Time-Delay Compensation in Environment Construction Using Laser Range Finder

Teleoperation is operating at a distance and thus extending human capabilities to performing tasks remotely by providing the operator with similar conditions as those at the remote location. Generally, operator should be supported in real-time with an accurate data about the teleoperation environment. This paper proposes a time-delay compensation method for real-time environment construction for vehicle teleoperation as a part of the work for modifying the well-developed Autonomous Ground Vehicle of Beijing Institute of Technology (BIT AGV) to be teleoperated using the Laser scanner data. The proposed Time-Delay Compensation Algorithm includes aprediction process of the vehicle position in the future time and an environment construction w. r. t. the predicted position when the vehicle is still in the current time. Results of the real and simulation experiments for the algorithm illustrate the simplicity in calculations, performance and effectiveness of the algorithm to compensate time-delay in environment construction for Vehicle Teleoperation.

Haptic interface for intuitive teleoperation of wheeled and tracked vehicles

In this paper, a novel haptic interface is proposed for general teleoperation of wheeled and tracked vehicles. The new mechanism of the proposed haptic interface, shown in Figure 1, not only allows human operators to easily teleoperate various types of target vehicles, including car-like vehicles, mobile robots, and tracked vehicles, but also improves the operator’s perception of the target vehicle’s operating status and its environment by introducing a “cornering feel” to the field of vehicle teleoperation. In addition, the proposed interface enables human operators to give commands to (or drive) the target vehicle in a way that is traditionally carried out by direct control or driving. Experiments were completed to carefully evaluate and test the performance of the proposed interface. The results show that the developed haptic master device is sufficient and suitable for general wheeled and tracked-vehicle teleoperation.

Force-Stiffness Feedback in Uninhabited Aerial Vehicle Teleoperation with Time Delay

This study investigates the use of haptic feedback to support the teleoperation of an uninhabited aerial vehicle with time delay. Two means of supplying the haptic feedback are investigated: first, an additional force on the control inceptor, and second, an additional force combined with an increased stiffness of the control inceptor. The advantage of combining the force feedback with stiffness feedback, over force feedback alone, is that the additional stiffness of the control inceptor increases the authority of the haptic feedback system in critical situations. The goal of this study is to increase the safety of teleoperation while reducing operator workload. A theoretical analysis shows that force― stiffness feedback improves the stability in the human control loop while allowing for lower force-feedback gain settings as compared with force feedback alone, which indicates that this could contribute to reducing operator workload. In an experiment, the two haptic feedback conditions were compared with a baseline condition without haptic feedback. The fidelity of the experiment was improved over an earlier experiment by introducing a time penalty for collisions. The number of collisions was different for all three conditions, with the lowest number for the force―stiffness condition and the highest number for the baseline condition. Both haptic feedback conditions were rated equally on subjective workload metrics and scored better than the baseline. This finding is in contrast with results from earlier experiments, which indicated a higher workload with haptic feedback, and shows the importance of creating realistic test conditions when using subjective workload ratings.

Haptic Feedback in Uninhabited Aerial Vehicle Teleoperation with Time Delay

In the teleoperation of an uninhabited aerial vehicle, haptic feedback can be used to provide useful information through the sense of touch. Although this can improve performance and enhance situation awareness, time delays caused by signal transmission generally induce control problems with haptic feedback, causing unsafe operations. Wave variables have been suggested to cope with the control problems in bilateral human-machine systems with time delay. Very little has been reported, however, on effects related to human control activity and workload when using wave variables. This paper describes a theoretical analysis of using wave variables to enhance a collision-avoidance system for visual and haptic uninhabited aerial vehicle teleoperation with time delay. An experiment was conducted to evaluate its effectiveness regarding operator performance, control activity, workload, and the safety of operation. Results indicate that wave variables are indeed successful in improving human performance. The number of collisions, operator control activity, and workload all decreased. Whereas performance and control activity were equivalent to the situation of haptic feedback without time delay, workload remained higher.

Master–Slave Control of a Teleoperated Anthropomorphic Robotic Arm With Gripping Force Sensing

This paper details the design and development of a low-cost control rig to intuitively manipulate an anthropomorphic robotic arm using a bilateral master-slave control methodology. Special emphasis has been given to the ease of operation and some form of force sensation. The control rig is fitted to the user's arm, and the forces exerted by the robotic arm's various joints are fed back to the user. Of special significance is the force feedback from the slave when its gripper is in contact with a real object. Several methods of force sensing have been explored and detailed. The effectiveness of the proposed method is confirmed by experiments on a commercially available robotic arm, which is controlled by a prototype three-axis master unit. The robotic arm mimics the dexterity of the human hand, wrist, and fingers. The proposed master control unit is cost effective and will have wide-ranging applications in the fields of medicine, manufacturing, security, extreme environment, entertainment, and remotely operated vehicle teleoperation in undersea recovery or extraterrestrial exploration vehicle