An NLP-based system for modulating virtual experiences using speech instructions

Abstract

The application of virtual reality (VR) technology in rehabilitation is a research area that has been attracting remarkable attention recently, especially as an intervention with learning and developmental disorders such as dyslexia and autism. Multi-modal VR-based rehabilitative systems can provide a rich experience to patients by responding to various types of bio-signals. This paper proposes a framework for incorporating voice input in rehabilitative games. The proposed framework equips collaborative learning environments with voice based interaction which allows for a more dynamic flow of therapy sessions between physically impaired patients and specialists in a rehabilitative VR environment. The proposed framework’s pipeline begins with a speech recognition module to parse utterances into sentences. Next, comes the feature extraction module to identify the intent of the user and entities as well. Finally, we modulate the 3D environment in Unity. The framework was designed in a way that guarantees smooth integration with any game or 3D environment. Several technologies were tested in terms of accuracy and speed to increase the overall performance of the pipeline. To the best of our knowledge, this research is the first to do the following: 1) build a dataset of instructions, inspired by a product already used in the healthcare market, 2) based on which we could realize a multi-modal system, with voice as one of the modalities, for the context of VR-based rehabilitation. We evaluate every stage in the system pipeline, and we show that we can achieve considerable results on each of these stages including the speech recognition module, with a 5.64% word error rate, and on intent classification where the BiLSTM model outperforms BERT in the Named Entity Recognition by 11% in terms of the F1-score. We envisage the proposed system to lend itself well to emerging healthcare frameworks in the Metaverse.