Simulating virtual electronic instrument with motion sensor

Abstract

Modern technology is changing every day. There is no longer only a keyboard mouse and a computer screen between the computer and the user. The research on human-computer interaction has gradually become an important field related to computer science and information engineering. A wide variety of sensors, video camera, and head-mounted displays greatly increase the way people communicate with computers. Today's environment is that everyone has a smart phone, so smart phones become the easiest device to develop human-computer interaction. In this study, we use the sensors inside the smart phone, gyroscope and accelerometer sensor, to track the direction, acceleration and the user's completed motion in 3D space. In addition, for the end user to provide a more realistic experience, we also use the Leap Motion controller. Thereby, we use these three body sensors to simulate electronic musical instruments and propose their system design, including drum sets, Tambourine and Theremin instruments. In the system, the web audio is connected to the sensor to emit sound effects. At the same time, the animation of the three-dimensional computer graphics is displayed by WebGL.