Automatic Transcription of Piano Music Using Audio-Vision Fusion

Abstract

Over the last decade many sophisticated and application-specific methods have been proposed for transcription of polyphonic music. However, the performance seems to have reached a limit. This paper describes a high-performance piano transcription system with two main contributions. Firstly, a new onset detection method is proposed using a specific energy envelope matched filter, which has been proved very suitable for piano music. Secondly, a computer-vision method is proposed to enhance audio-only piano music transcription, using the recognition of the player's hands on the piano keyboard. We carried out comparable experiments respectively for onset detection and overall system based on the MAPS database and the video database. The results were compared with the best piano transcription system in MIREX 2008, which still kept the best performance in piano subset till MIREX 2012. The results show that the system outperforms the state-of-art method substantially.