Automated assessment of Parkinsonian finger-tapping tests through a vision-based fine-grained classification model

Abstract

Movement disorder of Parkinson’s disease (PD) is usually quantified by the Movement Disorders Society-sponsored Revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) to evaluate its severity. However, the lack of well-trained experts and subjective inter-rater variability often limit an effective and objective assessment in clinical practice. Hence, developing an automated assessment method for movement disorders in PD is crucial. Here, we present a novel vision-based fine-grained action recognition model to cope with one of the most critical and challenging tasks in clinical scales: the finger-tapping test. Specifically, we establish a three-stream fine-grained classification network with a Markov chain fusion model to aggregate multi-stream information of the skeleton sequence from finger-tapping test videos. Then, we develop a spatial–temporal attention mechanism to capture rich spatial and temporal long-range dependencies from skeleton data and introduce a symmetric bilinear pooling layer to enrich the local feature representation of each stream’s output. Besides, a mini-batch-based balanced algorithm is designed to ensure that the samples in each mini-batch are inter-class balanced, thus mitigating the effect of imbalanced data on neural networks. Finally, our three-stream fine-grained classification network achieved an accuracy of 72.4% and an acceptable accuracy of 98.3% on 157 patients and 744 videos. Extensive experiments further confirm our approach’s effectiveness and reliability. This method does not require any wearable device and has excellent potential for remote monitoring of PD patients in the future.