Multiple Video Sequences Research Articles

Inverse Compositional Estimation of 3D Pose And Lighting in Dynamic Scenes

In this paper, we show how to estimate, accurately and efficiently, the 3D motion of a rigid object and time-varying lighting in a dynamic scene. This is achieved in an inverse compositional tracking framework with a novel warping function that involves a 2D --> 3D --> 2D transformation. This also allows us to extend traditional two frame inverse compositional tracking to a sequence of frames, leading to even higher computational savings. We prove the theoretical convergence of this method and show that it leads to significant reduction in computational burden. Experimental analysis on multiple video sequences shows impressive speed-up over existing methods while retaining a high level of accuracy.

Partitioning of Multiple Fine-Grained Scalable Video Sequences Concurrently Streamed to Heterogeneous Clients

Fine-grained scalable (FGS) coding of video streams has been proposed in the literature to accommodate client heterogeneity. FGS streams are composed of two layers: a base layer, which provides basic quality, and a single enhancement layer that adds incremental quality refinements proportional to number of bits received. The base layer uses nonscalable coding which is more efficient in terms of compression ratio than scalable coding used in the enhancement layer. Thus for coding efficiency larger base layers are desired. Larger base layers, however, disqualify more clients from getting the stream. In this paper, we experimentally analyze this coding efficiency gap using diverse video sequences. For FGS sequences, we show that this gap is a non-increasing function of the base layer rate. We then formulate an optimization problem to determine the base layer rate of a single sequence to maximize the average quality for a given client bandwidth distribution. We design an optimal and efficient algorithm (called FGSOPT) to solve this problem. We extend our formulation to the multiple-sequence case, in which a bandwidth-limited server concurrently streams multiple FGS sequences to diverse sets of clients. We prove that this problem is NP-Complete. We design a branch-and-bound algorithm (called MFGSOPT) to compute the optimal solution. MFGSOPT runs fast for many typical cases because it intelligently cuts the search space. In the worst case, however, it has exponential time complexity. We also propose a heuristic algorithm (called MFGS) to solve the multiple-sequence problem. We experimentally show that MFGS produces near-optimal results and it scales to large problems: it terminates in less than 0.5 s for problems with more than 30 sequences. Therefore, MFGS can be used in dynamic systems, where the server periodically adjusts the structure of FGS streams to suit current client distributions.

Tri-focal tensor-based multiple video synchronization with subframe optimization

In this paper, we present a novel method for synchronizing multiple (more than two) uncalibrated video sequences recording the same event by free-moving full-perspective cameras. Unlike previous synchronization methods, our method takes advantage of tri-view geometry constraints instead of the commonly used two-view one for their better performance in measuring geometric alignment when video frames are synchronized. In particular, the tri-ocular geometric constraint of point/line features, which is evaluated by tri-focal transfer, is enforced when building the timeline maps for sequences to be synchronized. A hierarchical approach is used to reduce the computational complexity. To achieve subframe synchronization accuracy, the Levenberg-Marquardt method-based optimization is performed. The experimental results on several synthetic and real video datasets demonstrate the effectiveness and robustness of our method over previous methods in synchronizing full-perspective videos.

Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score

We present a new variational method for multi-view stereovision and non-rigid three-dimensional motion estimation from multiple video sequences. Our method minimizes the prediction error of the sha...

TRACKING THE BALL AND PLAYERS FROM MULTIPLE FOOTBALL VIDEOS

This paper deals with tracking players and the ball in multiple soccer video sequences taken from fixed cameras located around the stadium. One of the major obstacle in player tracking occurs when occlusion between people results in insufficient observation information. If a player blob is not visible in some of the input videos due to occlusion, one solution is to check whether he is visible in others. This check is helped by the homography transformation induced by the play-ground because the players' positions can be represented on a virtual planar soccer field and can be mapped into any video image through a corresponding homography between the virtual field and the video image. Therefore, the position estimate of the player is exploited to predict and check the visibility in each of the video images. Measurements are taken from those visible images. On the other hand, before initiating tracking, we do a color correction in order to enhance the measurement and matching process because each of the video produces different RGB values for the same scene object due to different camera position and color sensor characteristics. After tracking players, ball tracking is carried out by eliminating the image blobs of players and accumulating the ideally ball-only images. We implemented our tracking algorithm, and experiments for sets of multiple real videos showed promising results.

A weighted probabilistic approach to face recognition from multiple images and video sequences

To date, advances in face recognition have been dominated by the design of algorithms that do recognition from a single test image. Recently, an obvious but important question has been put forward. Will the recognition results of such approaches be generally improved when using multiple images or video sequences? To test this, we extend the formulation of a probabilistic appearance-based face recognition approach (which was originally defined to do recognition from a single still) to work with multiple images and video sequences. In our algorithm, as it is the case in most appearance-based approaches, we will need to use a feature extraction algorithm to find those features that best describe and discriminate among face images of distinct people. We will show that regardless of the algorithm used, the recognition results improve considerably when one uses a video sequence rather than a single still. Hence, a positive answer to our question (in the general sense) seems reasonable. The probabilistic algorithm we propose in this paper is robust to partial occlusions, orientation and expression changes, and does not require of a precise localization of the face or facial features. We will also show how these problems are more easily solved when one uses a video sequence rather than a single image for testing. The limitations of our algorithm will also be discussed. Understanding the limitations of current techniques when applied to video is important, because it helps identify those weak points that require further consideration.

Photorealistic 3D reconstruction from handheld cameras

One of the major challenges in the fields of computer vision and computer graphics is the construction and representation of life-like virtual 3D scenarios within a computer. The VISIRE project attempts to reconstruct photo-realistic 3D models of large scenarios using as input multiple freehand video sequences, while rendering the technology accessible to the non-expert. VISIRE is application oriented and hence must deal with multiple issues of practical relevance that were commonly overlooked in past experiences. The paper presents both an innovative approach for the integration of previously unrelated experiences, as well as a number of novel contributions, such as: an innovative algorithm to enforce closedness of the trajectories, a new approach to 3D mesh generation from sparse data, novel techniques dealing with partial occlusions and a method for using photo-consistency and visibility constrains to refine the 3D mesh.

Network TV Broadcasting with Multi-Programs on Application-Oriented QoS

This paper introduces the design procedure of the contrived network TV broadcasting transcoder/encoder system, especially develops a new variable bit-rate (VBR) coding bit allocation strategy with a constraint channel bandwidth and consistent picture quality for multiple parallel video sequences broadcasting, which accommodates the complicated video sources with different frame rates and GOP structures, and combines the buffer control and the optimized macroblock (MB) coding mode selection. The proposed strategy absorbs several reasonable metrics in a hierarchical structure, and provides flexibility and promotion for resource allocation in multi-access scenario. Experimental results demonstrate the effect of the proposed scheme.

Virtual view synthesis of people from multiple view video sequences

This paper addresses the synthesis of novel views of people from multiple view video. We consider the target area of the multiple camera 3D Virtual Studio for broadcast production with the requirement for free-viewpoint video synthesis for a virtual camera with the same quality as captured video. A framework is introduced for view-dependent optimisation of reconstructed surface shape to align multiple captured images with sub-pixel accuracy for rendering novel views. View-dependent shape optimisation combines multiple view stereo and silhouette constraints to robustly estimate correspondence between images in the presence of visual ambiguities such as uniform surface regions, self-occlusion, and camera calibration error. Free-viewpoint rendering of video sequences of people achieves a visual quality comparable to the captured video images. Experimental evaluation demonstrates that this approach overcomes limitations of previous stereo- and silhouette-based approaches to rendering novel views of moving people.

A joint rate control scheme for H.264 encoding of multiple video sequences

The objective of joint rate control is to dynamically distribute the channel capacity among video sequences according to their respective complexities, thus a more uniform picture quality and a more efficient utilization of channel capacity are achieved. Most existing approaches are based on MPEG2 coding platform. This paper presents a novel joint rate control scheme for multiple video sequences coding based on H.264 standard. The joint rate control is performed hierarchically from a group of pictures of all sequences down to each picture of every sequence, depending on both the complexity of frames and the buffer occupancy. A novel complexity measure that adapts to the characteristics of H.264 video coding is proposed. An improved joint buffer control strategy, which is extended from that in the single sequence rate control scheme, is applied. Experimental results show that the proposed scheme maintains a good balance in picture quality among the sequences as well as within a sequence.

Space-time super-resolution

We propose a method for constructing a video sequence of high space-time resolution by combining information from multiple low-resolution video sequences of the same dynamic scene. Super-resolution is performed simultaneously in time and in space. By "temporal super-resolution," we mean recovering rapid dynamic events that occur faster than regular frame-rate. Such dynamic events are not visible (or else are observed incorrectly) in any of the input sequences, even if these are played in "slow-motion." The spatial and temporal dimensions are very different in nature, yet are interrelated. This leads to interesting visual trade-offs in time and space and to new video applications. These include: 1) treatment of spatial artifacts (e.g., motion-blur) by increasing the temporal resolution and 2) combination of input sequences of different space-time resolutions (e.g., NTSC, PAL, and even high quality still images) to generate a high quality video sequence. We further analyze and compare characteristics of temporal super-resolution to those of spatial super-resolution. These include: How many video cameras are needed to obtain increased resolution? What is the upper bound on resolution improvement via super-resolution? What is the temporal analogue to the spatial "ringing" effect?

A Knapsack Model for Bandwidth Management of Prerecorded Multiple MPEG Video Sources

In this article we provide a framework for controlling the bit rate of multiple prerecorded MPEG video sequences by choosing the quantization factors assigned to individual sources in a way that the total mean square error at the output of the encoder is minimized. We propose and test a knapsack model for the selection of the quantization factors. Our computations based on a set of relatively diverse video sequences reveal that the proposed model achieves a high utilization of the available bandwidth and acceptable distortion levels without any data loss.

Spatio-temporal alignment of sequences

This paper studies the problem of sequence-to-sequence alignment, namely, establishing correspondences in time and in space between two different video sequences of the same dynamic scene. The sequences are recorded by uncalibrated video cameras which are either stationary or jointly moving, with fixed (but unknown) internal parameters and relative intercamera external parameters. Temporal variations between image frames (such as moving objects or changes in scene illumination) are powerful cues for alignment, which cannot be exploited by standard image-to-image alignment techniques. We show that, by folding spatial and temporal cues into a single alignment framework, situations which are inherently ambiguous for traditional image-to-image alignment methods, are often uniquely resolved by sequence-to-sequence alignment. Furthermore, the ability to align and integrate information across multiple video sequences both in time and in space gives rise to new video applications that are not possible when only image-to-image alignment is used.

Bit-rate control for uniform picture quality within composed video sequence

In a continuous presence multipoint control unit, the multiple video sequences of the high spatial resolution are decoded and downconversed, then composed into an output video sequence. When the bandwidth to transmit the composed sequence is fixed, if the bit rate for each low spatial resolution sequence within the composed sequence is independently allocated, then the degradation of subjective picture quality among the low spatial resolution sequences is severe due to the different video characteristics. An adaptive bit-rate control method to maintain the uniform picture quality among the low spatial resolution sequences is presented. We first find a relationship of bit rate and quantizer parameter for the video manipulation and composition environments. Then we obtain the bit rate for each low spatial resolution sequence to have same quantizer parameter among the sequences by adaptively using the coded results of low or high spatial resolution sequences while considering the scene change of each sequence. We show by simulation that the proposed adaptive bit-rate control method can maintain almost uniform picture quality among the low spatial resolution sequences and almost constant picture quality within the composed sequence in comparison to an independent bit-rate control method.

Reality modeling and visualization from multiple video sequences

A reality modeling and visualization system called Immersive Video uses multiple videos of an event from different perspectives to generate a 3D digital sequence of object movement. Given appropriate camera coverage, full 3D digital videos can be generated using today's technology. The potential for photorealistic videos from arbitrary perspectives exists, although the coverage and degree of realism in virtual views will ultimately be determined by the application. Many issues demand resolution before one will see practical and realistic 3D digital video, including more robust shape recovery methods, broadcast-quality virtual views, and real-time interactive rendering of long 3D sequences.