Inserting synthetic characters into live-action scenes of multiple people

Abstract

Using a single camera, we capture video of multiple people walking in an open area and insert a moving synthetic character into the scene. The people are located in the video using simple image processing techniques. Simplifying assumptions and basic projective geometry are used to locate the people in 3D. We extract sufficient information to synthetically reconstruct the 3D scene including the paths of individuals in the crowd. These paths are also used to plan the paths of new synthetic characters avoiding collisions with the existing people. Lighting samples taken from the real environment and modeled in the synthetic scene are used along with camera calibration specifications. The new synthetic characters are rendered in 2D and composited into the real video using transparency alpha values extracted by the algorithm. Because we reconstruct the scene in 3D, the new synthetic character is occluded by, and occludes, the people in the original video. This is accomplished with minimal user interaction.