An interactive multimedia storybook demonstration system

Abstract

Different from a real, physical storybook that shows texts and still pictures only, we present an Interactive Multimedia Storybook Demonstration System. The objective is to develop an automatic system through which users can browse digital multimedia by turning pages of a physical storybook, scribbling with an infrared pen, and interacting with a virtual keyboard. Our system supports the three functionalities: (a) page number recognition; (b) infrared interaction; and (c) virtual keyboard. Our system is vision-based using digital webcams and infrared cameras as input devices and a projector for displaying digital multimedia. The method is based on finite-state machines with a combination of computer vision techniques using C/C++ programming and the OpenCV library, resulting in a relatively low-cost and near real-time interactive system. For the implementation, our system incorporates a projector, two digital webcams, an infrared camera, an interactive platform (i.e., the storybook), and a personal computer. Results of the system functionalities are shown as examples during the demonstration. In addition, pilot studies were performed for system evaluation indicating that our system could achieve high recognition rates of over 90 % in near real time (system response time <0.2 s). The usability evaluation indicated that our system could yield an effective interaction for the three functionalities, although minor lag may be observed during the infrared interaction. In summary, our system has been successfully demonstrated in a large exhibition with over hundreds of user experience and interaction, which clearly shows the potential of our system in numerous applications for storybooks and the like.