Evaluation of Subjective and Objective Performance Metrics for Haptically Controlled Robotic Systems

Pål J. From,Cong Dung Pham,Huynh Nhat Trinh Phan

doi:10.4173/mic.2014.3.2

Abstract

This paper studies in detail how dierent evaluation methods perform when it comes to describing the performance of haptically controlled mobile manipulators. Particularly, we investigate how well subjective metrics perform compared to objective metrics. To find the best metrics to describe the performance of a control scheme is challenging when human operators are involved; how the user perceives the performance of the controller does not necessarily correspond to the directly measurable metrics normally used in controller evaluation. It is therefore important to study whether there is any correspondence between how the user perceives the performance of a controller, and how it performs in terms of directly measurable metrics such as the time used to perform a task, number of errors, accuracy, and so on. To perform these tests we choose a system that consists of a mobile manipulator that is controlled by an operator through a haptic device. This is a good system for studying dierent performance metrics as the performance can be determined by subjective metrics based on feedback from the users, and also as objective and directly measurable metrics. The system consists of a robotic arm which provides for interaction and manipulation, which is mounted on a mobile base which extends the workspace of the arm. The operator thus needs to perform both interaction and locomotion using a single haptic device. While the position of the on-board camera is determined by the base motion, the principal control objective is the motion of the manipulator arm. This calls for intelligent control allocation between the base and the manipulator arm in order to obtain intuitive control of both the camera and the arm. We implement three dierent approaches to the control allocation problem, i.e., whether the vehicle or manipulator arm actuation is applied to generate the desired motion. The performance of the dierent control schemes is evaluated, and our findings strongly suggest that objective metrics better describe the performance of the controller, even though there is a clear correlation between subjective and objective performance metrics.

Highlights

Teleoperation and haptic control allow operators to control remotely located objects from a safe and comfortable location
The performance of the different control schemes is evaluated, and our findings strongly suggest that objective metrics better describe the performance of the controller, even though there is a clear correlation between subjective and objective performance metrics
In this paper we have compared subjective and objective performance metrics for evaluating the performance of different controllers for mechanical systems that are haptically controlled by human operators

Summary

Introduction

Teleoperation and haptic control allow operators to control remotely located objects from a safe and comfortable location. The main motivation for remotely operated robots is to relieve humans from entering hostile and dangerous environments and to utilize robots in areas where humans do not have access This kind of systems poses several challenges when it comes to the evaluation of the control scheme, as it is the directly measurable metrics that define the performance of the controller, and how the operator doi:10.4173/mic.2014.3.2 c 2014 Norwegian Society of Automatic Control. All teleoperated systems have one thing in common; they are controlled by a human operator, so how the human operator perceives the controller performance should be an important criterion when designing the controller It is not straightforward, to find suitable metrics for this kind of subjective performance evaluations. It is interesting to investigate whether or not the actual performance of the system is reflected in the feedback that we get from the operator

Methods

Results

Conclusion