Realizing computer vision rehabilitation assessment tests & evaluation applications for mobile devices

Abstract

Upper extremity impairments are a common consequence of stroke, necessitating thorough rehabilitation monitoring and kinematic assessments to facilitate motor recovery. The Box and Block Test (BBT) and Sollerman Hand Function Test (SHFT) are two widely utilized and recommended tools for objectively measuring upper limb dexterity and evaluating fine motor skill rehabilitation in patients. However, these tests rely on specific equipment and therapist attendance, making the process time-consuming and clinic-dependent. This paper introduces a computer vision-based hand rehabilitation assessment suite specifically designed for mobile devices, such as smartphones and tablets, which serves as a virtual alternative to traditional methods while also incorporating an interactive exergame. Our application faithfully integrates the original tests’ guidelines and procedures into an engaging computer vision experience, utilizing advanced technologies like MediaPipe Hands for precise hand and finger tracking. This innovative solution obviates the need for additional computer peripherals such as smart gloves or VR headsets, as well as physical equipment like wooden boxes and blocks, relying solely on the built-in camera of everyday mobile devices. In addition, we address several technical challenges encountered in our approach and outline future directions for score normalization and feature expansion, ensuring the continued improvement and efficacy of our hand rehabilitation assessment suite.