HFR-Video-Based Fingertip Velocimeter for Multifinger Tapping Detection

Abstract

In this study, we propose a novel concept of a software-based fingertip velocimeter using high-frame-rate (HFR) video processing that can simultaneously estimate when and where an operator taps with his/her finger by detecting the high-frequency component that develops when the fingertip actively contacts something. Our software-based fingertip velocimeter can precisely estimate the velocities of multiple fingers through HFR video processing in real time. Digital image correlation (DIC) operating at every frame for sub-pixel-precision velocity estimation is hybridized with CNN-based object detection operating at intervals of dozens of frames to robustly update the fingertip ROI regions during the frame-by-frame DIC operation. We developed a real-time multi-finger tapping detection system that can execute DIC operation on 720×540 resolution images at 500 fps with CNN-based fingertip detection at 30 fps. By presenting several experimental results for finger tapping detection, including virtual keyboard interaction with a ten-finger keyboard input, the effectiveness of our fingertip velocimeter as a finger tapping interface was demonstrated, which can simultaneously estimate the tapping positions and moments of multiple fingers when finger tapping is performed 10 times or more in a second.