Application of multimodality perception scene construction based on Internet of Things (IoT) technology in art teaching

Abstract

Numerous impediments beset contemporary art education, notably the unidimensional delivery of content and the absence of real-time interaction during instructional sessions. This study endeavors to surmount these challenges by devising a multimodal perception system entrenched in Internet of Things (IoT) technology. This system captures students’ visual imagery, vocalizations, spatial orientation, movements, ambient luminosity, and contextual data by harnessing an array of interaction modalities encompassing visual, auditory, tactile, and olfactory sensors. The synthesis of this manifold information about learning scenarios entails strategically placing sensors within physical environments to facilitate intuitive and seamless interactions. Utilizing digital art flower cultivation as a quintessential illustration, this investigation formulates tasks imbued with multisensory channel interactions, pushing the boundaries of technological advancement. It pioneers advancements in critical domains such as visual feature extraction by utilizing DenseNet networks and voice feature extraction leveraging SoundNet convolutional neural networks. This innovative paradigm establishes a novel art pedagogical framework, accentuating the importance of visual stimuli while enlisting other senses as complementary contributors. Subsequent evaluation of the usability of the multimodal perceptual interaction system reveals a remarkable task recognition accuracy of 96.15% through the amalgamation of Mel-frequency cepstral coefficients (MFCC) speech features with a long-short-term memory (LSTM) classifier model, accompanied by an average response time of merely 6.453 seconds—significantly outperforming comparable models. The system notably enhances experiential fidelity, realism, interactivity, and content depth, ameliorating the limitations inherent in solitary sensory interactions. This augmentation markedly elevates the caliber of art pedagogy and augments learning efficacy, thereby effectuating an optimization of art education.