Continuous motion finger joint angle estimation utilizing hybrid sEMG-FMG modality driven transformer-based deep learning model

Abstract

Continuous motion joint angle estimation plays an important role in human–machine interaction (HMI). However, it is still a challenge to estimate continuous motion finger joint angles (CMFJA) accurately. To improve CMFJA estimation accuracy, we used hybrid surface electromyography-force myography (sEMG-FMG) modality as the decoding scheme since combining two sensing modalities could potentially compensate and correct for single sensing modality, and proposed a biosignals driven convolution neural networks (CNN) and Transformer model (BioCNN-T) to estimate CMFJA motivated by the potential of the Transformer architecture, which extracts local features through CNN, and captures dependencies between global features through Transformer encoder. The metacarpophalangeal joint angles of 6 hand movements commonly used in daily life were selected as estimation objects. The experimental results show that CMFJA estimation by hybrid sEMG-FMG modality can achieve higher accuracy than single modality. And we found that the addition of convolution block before Transformer encoder plays a positive role in improving estimation accuracy and pixel-level fusion is the most suitable among the information fusion strategies appearing in the paper. What’s more, compare with the CNN and long short-term memory neural networks (CNN-LSTM), the BioCNN-T has higher accuracy and less computing cost. (Average normalized root mean square error, 0.0456 ± 0.0030 versus 0.0539 ± 0.0035; Inference time, 9.12 × 10-6 s versus 3.20 × 10-4 s). To our knowledge, our work is a pioneer in the use of hybrid sEMG-FMG modality and Transformer architecture to estimate CMFJA. The Implications of it have promising potential in flexible and fine-grained HMI.