AI System Design for Robotic Hand to Play the Piano

Abstract

Robotic and Artificial Intelligent (AI) have been introduced as a key factor for industry revolution4.0. Many industries, such as manufacturing, agriculture, logistics, supply chain, and so on, are transformed and applied robotic and AI to enhance productivity and reduce cost. AI in creative work is very challenging, especially in music. This paper presents a system to enable the robotic arm to play piano notes with minimal errors. We used the knowledge of Optical music recognition (OMR), Automatic music transcription (AMT), Music source separation (MSS), and the elimination of robot arm cycle times problems for creating this system. The robotic arm used in testing with this system was LEGO. It can perform 4 functions. The first function is to play piano notes from sheet piano using Sheet Vision and Tesseract-OCR. Note reading accuracy is 21.36%, and note reading accuracy with note duration is 13.46%. The second function is to play the piano like a piano sound separate from a music file using Spleeter and Onsets and Frames. Piano note accuracy is 58.32%, piano note onset time error is ± 0.17, and piano note duration time error is ± 0.65. The third function is to play piano notes from real-time piano sounds using Onsets and Frames Real-time mode. The last function is to play the piano notes from the brain waves by comparing the frequency of the brain waves with the frequency of the piano notes. The design and experimental results are explained in this paper.