JPL BioSleeve for gesture-based control: Technology development and field trials

Abstract

The JPL BioSleeve is a wearable gesture-based human interface for natural robot control. Activity of the user's hand and arm is monitored via surface electromyography sensors and an inertial measurement unit that are embedded in a forearm sleeve. Gesture recognition software then decodes the sensor signals, classifies gesture type, and maps the result to output commands to be sent to a robot. The BioSleeve interface can accurately and reliably decode as many as sixteen discrete hand and finger gestures and estimate the continuous orientation of the forearm. Here we report development of a new wireless BioSleeve prototype that enables portable field use. Gesture-based commands were developed to control a QinetiQ Dragon Runner tracked robot, including a 4 degree-of-freedom manipulator and a stereo camera pair. Gestures can be sent in several modes: for supervisory point-to-goal driving commands, virtual joystick for teleoperation of driving and manipulator, and pan-tilt of the camera. Hand gestures and arm positions are mapped to various commands recognized by the robot's onboard control software, and are meant to integrate with the robot's perception of its environment and its ability to complete tasks with various levels of autonomy. The portable BioSleeve interface was demonstrated through control of the Dragon Runner during participation in field trials at the 2014 Intuitive Robotic Operator Control Challenge. The successful completion of Challenge events demonstrated the versatility of the system to provide multiple commands in different modes of control to a robot operating under difficult real-world environmental conditions.