Human-computer interaction and animation system for simple interfacing to virtual environments

Abstract

Presents MARTI (Man-machine Animation Real-Time Interface) for simple interfacing to virtual environments. The system is designed for automated special-effect animation and human-computer interaction, and can immerse people, irrespective of their expertise, into new technologies. Previous research has used image recognition systems to extract control signals for lip synchronisation; however, the analysis requires that the key articulatory points be highlighted. Researchers have also considered acoustic soundtracks. However, these systems are limited by the accuracy of their speech recognition and must be trained on the single animator's voice. Furthermore, they do not provide timing information, which prevents accurate synchronisation. MARTI overcomes these limitations, allowing automatic lip synchronisation from a single speech input, and without the normal constraints of head-sets, reflectors and complex puppeteer control hardware. The system achieves a lip synchronisation performance in excess of 81% and does not require pre-training to the performer's voice, but instead operates with continuous speech in 'normal' non-laboratory conditions. Furthermore, the system returns timing information, and is invariant to regional accents and dialects, race, age and gender. MARTI introduces novel research from a number of engineering fields in order to realise the first natural interface and animation system to be capable of high performance for real users and real-world applications.