Bringing Gaming; VR; and AR to Life with Deep Learning

Abstract

Game development is a complex and labor-intensive endeavor. Game environments, storylines, audio, and character behaviors are carefully crafted requiring graphics artists, storytellers, and software developers to work unison. Often games end up with a delicate mix of hard-wired behavior the form of traditional code and somewhat more responsive behavior the form of large collections of complex rules. Similarly, audio, video, and graphics are carefully and manually curated and synchronized with game actions. The addition of Virtual Reality (VR) and Augmented Reality (AR) have only added to the many challenges that game developers and storytellers face. As a technology provider, it has always been our primary goal to democratize game development and digital storytelling. One way to do so is to to bring the latest technologies to our developers to help them overcome some of the above-mentioned challenges. Over the past few years, data intensive Machine Learning (ML) solutions have obliterated rule-based and manually curated systems the enterprise - think Amazon, Netflix, and Uber. Obviously, our developers operate a very different field from these enterprises, yet we have found great promise the data-driven ML approach to game development. ML and more specifically Deep Learning (DL) allow content creators to shift to data-driven approaches. DL has recently gained significant traction the research community the areas of Computer Vision and Speech Recognition. Success these areas have spurred the pursuit of Deep Reinforcement Learning (Deep RL) for gameplay and character development. Deep RL allows us to train Non-player Characters (NPC) rather than program them. NPC behavior can be learned either through human gameplay (Imitation Learning) or automated game simulations. Either way, as the complex and strategic behavior of an NPC can be learned, it potentially disrupts the way games are developed and perhaps even experienced by the consumer. Deep RL is not limited to game strategy and path finding as researchers also have demonstrated promising results the area of character animation. Another new area of interest for content creators is Generative Adversarial Nets (GANs). GANs are DL-based models that have learned to generate output that is representative of the input samples used to train them. In other words, these models can be used to generate very naturally looking textures such as grass or rocks. GANs can also be used to upscale low-resolution images by filling in the missing details. At Unity Technologies, we have explored the use of DL and GANs content creation and Deep RL character development. I will share our learnings and the Unity APIs we use with the audience and hopefully inspire content developers and researchers alike to start exploring these new technologies to create digital experiences that are out of this world.