Implementation of escape room system based on augmented reality involving deep convolutional neural network

Abstract

Escape room is a live-action adventure game, where the players search clues, solve puzzles and achieve the assigned tasks. This paper proposed a novel escape room system combining augmented reality and deep learning technology. The system adopts a client–server architecture and can be divided into the server module, the smart glasses module and the player–hardware interaction module. The player–hardware interaction module consists of subsystems each of which includes a Raspberry Pi 3. HoloLens is used as the smart glasses in the experiment of the paper. The server communicates with all the Raspberry Pis and HoloLens through TCP/IP protocol and manages all the devices to achieve the game flow by following the process timeline. The smart glasses module provides two display modes, i.e., the AR 3D models display and the 2D text clues display. In the first mode, the SDK Vuforia is used for detection and tracking of markers. In the second mode, the scene images captured by HoloLens camera are sent to the pre-trained image classifier based on deep convolutional neural network. Considering both the image category and the game status value, the server decides the text clue image to be displayed on HoloLens. The accuracy of the image classification model reaches 94.9%, which can be correctly classified for a certain rotation angle and partial occlusion. The integration of AR, deep learning, electronics and escape room games opens up exciting new directions for the development of escape room. Finally, a built mini-escape room is analyzed to prove that the proposed system can support more complicated narratives showing the potential of achieving immersion.