A Camera-Based Music-Making Tool for Physical Rehabilitation

Abstract

The universality of music among human cultures as well as our common experience of naturally responding to music with motion, seem to be widely recognized (Tramo 2001). Recent brain-imaging studies (Lahav et al. 2005; Lahav, Saltzman, and Schlaug 2007) show that humans, given appropriate auditory inputs, seem to be tuned to produce corresponding motor outputs. This unique auditorymotor interplay provides the conceptual basis for the use of music therapy, in particular, active music therapy, where a patient is physically involved in producing music rather than simply reacting to or accompanying music (Pacchetti et al. 2000; Paul and Ramsey 2000; Lahav 2005). Music therapy has benefited people with physical disabilities, mental health needs, developmental and learning disabilities, Alzheimer's and Parkinson's disease, autism, substance abuse problems, brain injuries, and acute and chronic pain (Pacchetti et al. 2000; Harris and Summa-Chadwick 2005; see also the American Music Therapy Association's Web site, www.musictherapy.org). Playing an instrument, such as piano, guitar, or drums, may be very difficult or even inf easible for patients with motor dysfunctions. As an alternative, easy-to-use tool for active music therapy, we designed Music M ker, a human-computer interface that converts body movements into musical and visual feedback in real time using the open software platform EyesWeb (Camurri et al. 2004). Music Maker is a non-obtrusive camera-based tool that allows physically impaired patients to naturally create music with no prior musical training. Patients simply move their hand or foot in space or on a surface, and Music Maker detects and interprets these movements. Detection is accomplished without any sensors or markers that must be attached to the patients' bodies. Music Maker relies only on the video input from a camera to observe patient motion and on computervision techniques to analyze the motion. Music Maker is an adaptive interface that can be adjusted to provide auditory and visual feedback based on the patient's needs and interests. Auditory feedback could range from a single piano note to a recording of the patient's favorite piece of music. Visual feedback is provided by a graphical display on a computer monitor or wall-mounted screen. Music Maker uses fun displays, for example, cartoon drawings or pictures of musical instruments. Its hardware setup can be adjusted according to patients' levels of impairment, their particular therapeutic goals, and the equipment available in hospitals or patients' homes. (See Figure 1 for sample setups.) Computer Music Journal, 31:2, pp. 39-53, Summer 2007 © 2007 Massachusetts Institute of Technology.