Lights! Dance! Freeze!

Abstract

Conventionally, spectators enjoy films passively. This paper describes an interactive film installation that invites participants to use their entire body as a query to search and explore a large corpus of musical films in a novel way. Using an RGB camera, ML-based skeleton tracking technology and a unique pose and film indexing system, this installation tracks a participant's movements and mirrors them in real-time by finding matching poses among hundreds of thousands from well-known musicals. When the participant freezes in a pose, the installation instantly plays back the short film clip that begins with that pose, immersing them in the music and dance from musicals of different eras. This approach explores themes of tangible interfaces and the new possibilities that emerge from employing embodied interaction to traverse the dance pose space, which is traditionally difficult to index and interact with in real time. The pose indexing system and whole-body interaction we propose in this paper open new pathways for cultural participation, as they lend themselves to different datasets and require no technical skills from participants.