Physics-Based Table-Top Mixed Reality Games

Abstract

Mixed reality applications use techniques from computer vision, augmented reality and virtual reality to allow real and virtual objects interact physically together on a user’s computer screen. This paper will describe two mixed reality applications which allow the user to play games that appear to take place on top of their physical desk. The games described are a desktop racing game and a desktop based game of ten pin bowling. In the desktop racing game virtual cars, controlled by the user, interact with both virtual objects (such as trees, walls and lampposts) and real ones (such as ramps and blocks). In the bowling game the player throws a real ball at a set of virtual bowling pins which react realistically as the ball appears to hit them. These initial games are being used as a springboard to investigate the core competencies required to make mixed reality games. Introduction Drawing on techniques from the fields of computer vision, augmented reality and virtual reality, mixed reality applications (Milgram et al, 1994) bring together real and virtual objects in applications where they interact closely together. This offers potential unique interactions not possible in other applications. One area where these potentials sit ready to be exploited is that of computer games. Recent innovations in computer game control techniques such as the Sony EyeToy and the Nintendo Wii Remote show the potential for such novel interaction paradigms. Mixed reality techniques could free game players from the confines of traditional game controllers making possible some very interesting possibilities. Figure 1: Screenshots of the two mixed reality physics-based table top games described in this paper 1 http://www.eyetoy.com/ 2 http://www.nintendo.com/wii/ This paper will describe two mixed reality games in development at the DIT Experimental Games Group (EGG) that are played on the physical desktop of a user and mix together real and virtual objects to create interesting game-play possibilities. The first of these games is a driving game in which the player drives a car around on the top of their desk performing stunts using real and virtual obstacles. The second game is a mixed reality bowling game in which the player throws a real bowling ball at a virtual set of bowling pins. Screenshots from both of these games are shown in figure 1. The next section of this paper will gave a background to mixed reality applications, including some existing games. The section after this will then describe our games, including a description of the technologies used and the lessons learned in developing them. Finally, some conclusions as to interesting avenues in which to take mixed reality applications will be drawn along with some suggestions as to what we intend to do next. Mixed Reality Environments In one of the earlier works on the subject, Milgrim et al (1994) define a mixed reality environment as “one in which real world and virtual world objects are presented together”. However, this leaves open a very large number of possibilities and so Milgrim et al propose a spectrum of mixed reality environments (reproduced in figure 2). This spectrum places real environments on one end, totally virtual environments on the other, and various blends of mixed reality environments across the remainder of the spectrum. Major classes of applications in this range include augmened reality (AR), in which a predominantly real environment is augmented with virtual objects (Bimber & Raskar, 2005, Green et al, 2008), and augmented virtuality (AV) in which a predominantly virtual envrionment is augmented with real ojects (Regenbrecht et al, 2003, Wang, 2007). Figure 2: Milgrim’s mixed reality spectrum The applications described in this work fit squarely at the centre of this spectrum, rather than towards either end. This is due to the fact that they allow such close interaction between virtual and real objects. Virtual objects are physically modelled and are affected by real ones – for example a virtual car crashing into a real obstacle, or a real bowling ball knocking over a virtual bolwing pin. In order to implement a mixed reality application there are a number of technical challenges that must be overcome. While this paper will not discuss any of these in detail it is worth mentioning the most important ones. Firstly, in order to allow real and virtual objects interact believably with one-another accurate tracking of real objects must be achieved. There is a large body of work from the field of computer vision on how to achieve this (Comaniciu, et al, 2003, Trucco & Plakas, 2006), but it is worth noting that often simple “cheats” are used such as only tracking black objects against white backgrounds or using special markers. Related to this first issue is the issue of registering real and virtual environments. This is one of the main concerns of augmented reality research. One of the most common solutions is the use of fiducial markers (such as the black and white square patterns shown in figure 1) which can be easily recognized from camera images and allow a transformation from the camera position to the marker position to be calculated. This is 3 http://www.seriousgames.ie/ important so that virtual objects appear in the correct positions and appear correctly aligned to real objects. Modelling the physical interactions between virtual and physical objects is another key challenge. This is achieved by giving real objects a virtual alias within the physics simulation of a scene, which can then cause interactions with virtual objects. One of the key challenges here is making these interactions appear believable, and this generally involves a large amount of trial and error based tweaking of parameters such as mass and density. Finally, the virtual objects in a scene must be rendered in such a way as to make it appear that they exist seamlessly alongside the real objects in the scene. Shadowing, occlusion and rendering techniques can be used for this. Without achieving each of these it is not possible to create a successful game of the type that we are looking to make. There has, though, been some very interesting work in the area and some very interesting games have been created. Mixed Reality Games One of the most exciting mixed reality games developed in recent years has been ARQuake (Thomas, 2003), an AR game that extends the famous computer game Quake. In the original Quake the player takes on the role of a soldier in a game played from a first person perspective and controlled using a combination of a keyboard and mouse. ARQuake used AR technology to bring Quake into the real world. Players can see virtual monsters in their real environment through a see-through head-mounted-display (HDM) (figure 3 (A)). Control is based on the player moving through their real environment and using a special gun shaped controller. An example of the ARQuake experience is shown in figure 3 (B). ARQuake is notable for its ambition to bring virtual objects into such a freeform and large environment.