Hands and Fingers Tracking for Tactile Graphics Reading Assistive Device

Abstract

AbstractAwareness on the difficulties faced by the blind and visually impaired (BVI) people to read tactile graphics has caught the attention of many researchers to develop assistive devices using machine vision approach. At the moment, most techniques that have been used to detect and track hands and fingers are insufficient to support the active and complex behavior of the tactile graphics reading and exploration. This paper presents a system to track both hands and multiple fingers using MediaPipe Hands solution. A benchmark experiment has shown the accuracy of the system to detect and track for single and both hands with 93.1 and 99.9% respectively at an average speed of 20 FPS. Besides that, a preliminary design of tactile graphics reading assistive device has been introduced in this paper. By using the xy-coordinates that have been extracted from the hand landmarks detection, a specific hand gesture and conditions have been configured to allow reader to interact with the tactile graphics. The works presented have shown a promising result to be further explored for future development of tactile graphics reading assistive device which support two hands and multiple fingers exploration.KeywordsVisual impairmentBlindnessTactile graphicsTactile explorationHand and finger tracking systemMediaPipe