Synchronized Video Streams Research Articles

Development of a universal binary classifier of the state of artillery barrels by the physical fields of shots

An artillery shot is accompanied by the release of combustion products of powder gases from the barrel. It is proposed to usemuzzle ejection to classify the level of barrel wear during firing. A full-scale experiment was carried out with high-speed video recording in the visible and infrared ranges of the dynamics of the development of muzzle ejection when firing guns with a defect-free and worn barrel. Muzzle ejection when fired from a large-caliber gun consists of three spatial regions: frontal and two side, associated with the emission of powder gases through the openings of the compensators. A technique for analyzing three synchronized video streams has been developed. The technique made it possible to quite fully investigate the processes of muzzle ejectiondevelopment dynamics in defect-free and worn barrels. Informative signs are chosen, which are different for the dynamics of muzzle ejection from defect-free and worn barrels. This madeit possible to build a binary classifier of the condition of the trunks by the level of wear based on the support vector machine with least squares. In contrast to the classical SVM classifier, thisallowed us to reduce the calculation time and reduce the required size of the training set. To assess the quality of classification, it is proposed to rely on only errors of the first and second kind, but also an integral indicator –the probability of error-free classification. To increase the reliability of the classification, the concept of a universal binary classifier is proposed, which uses both video recording of the muzzle ejection and acoustic fields of the shot –ballistic and muzzle waves –to diagnose the state of the barrel. On the basis of experimental data, it is shown that the use of all physical fields accompanying an artillery shot for the binary SVM classification allows obtaining a high value of the error-free classification probability.

Towards real-time 3D visualization with multiview RGB camera array.

A real-time 3D visualization (RT3DV) system using a multiview RGB camera array is presented. RT3DV can process multiple synchronized video streams to produce a stereo video of a dynamic scene from a chosen view angle. Its design objective is to facilitate 3D visualization at the video frame rate with good viewing quality. To facilitate 3D vision, RT3DV estimates and updates a surface mesh model formed directly from a set of sparse key points. The 3D coordinates of these key points are estimated from matching 2D key points across multiview video streams with the aid of epipolar geometry and trifocal tensor. To capture the scene dynamics, 2D key points in individual video streams are tracked between successive frames. We implemented a proof of concept RT3DV system tasked to process five synchronous video streams acquired by an RGB camera array. It achieves a processing speed of 44 milliseconds per frame and a peak signal to noise ratio (PSNR) of 15.9 dB from a viewpoint coinciding with a reference view. As a comparison, an image-based MVS algorithm utilizing a dense point cloud model and frame by frame feature detection and matching will require 7 seconds to render a frame and yield a reference view PSNR of 16.3 dB.

Wide baseline pose estimation from video with a density-based uncertainty model

Robust wide baseline pose estimation is an essential step in the deployment of smart camera networks. In this work, we highlight some current limitations of conventional strategies for relative pose estimation in difficult urban scenes. Then, we propose a solution which relies on an adaptive search of corresponding interest points in synchronized video streams which allows us to converge robustly toward a high-quality solution. The core idea of our algorithm is to build across the image space a nonstationary mapping of the local pose estimation uncertainty, based on the spatial distribution of interest points. Subsequently, the mapping guides the selection of new observations from the video stream in order to prioritize the coverage of areas of high uncertainty. With an additional step in the initial stage, the proposed algorithm may also be used for refining an existing pose estimation based on the video data; this mode allows for performing a data-driven self-calibration task for stereo rigs for which accuracy is critical, such as onboard medical or vehicular systems. We validate our method on three different datasets which cover typical scenarios in pose estimation. The results show a fast and robust convergence of the solution, with a significant improvement, compared to single image-based alternatives, of the RMSE of ground-truth matches, and of the maximum absolute error.

Line-preserving video stitching for asymmetric cameras

The narrow FOV on mobile phones is becoming unsatisfying with the development of video live broadcast. Fast and stable video stitching algorithm is urgently needed with the advent of multi-device bracket and other equipment. In this paper, we propose a novel stitching method to stitch multiple synchronized video streams which are captured from different fixed cameras. This procedure can be divided into two stages: initial template construction and seam updating. In the stage of initialization, we align the extracted background with proposed line-preserving alignment, and save the mesh for future stitching. Especially, scale estimation allows the algorithm to be applied to videos from asymmetric cameras of different resolutions and scales. In the seam updating stage, we propose a last-frame-guided seam updating method to find a smooth transition on spatial-temporal volumes. Experimental results demonstrate that our method can efficiently stitch videos without ghosting and artifacts.

Panoptic Studio: A Massively Multiview System for Social Interaction Capture.

We present an approach to capture the 3D motion of a group of people engaged in a social interaction. The core challenges in capturing social interactions are: (1) occlusion is functional and frequent; (2) subtle motion needs to be measured over a space large enough to host a social group; (3) human appearance and configuration variation is immense; and (4) attaching markers to the body may prime the nature of interactions. The Panoptic Studio is a system organized around the thesis that social interactions should be measured through the integration of perceptual analyses over a large variety of view points. We present a modularized system designed around this principle, consisting of integrated structural, hardware, and software innovations. The system takes, as input, 480 synchronized video streams of multiple people engaged in social activities, and produces, as output, the labeled time-varying 3D structure of anatomical landmarks on individuals in the space. Our algorithm is designed to fuse the "weak" perceptual processes in the large number of views by progressively generating skeletal proposals from low-level appearance cues, and a framework for temporal refinement is also presented by associating body parts to reconstructed dense 3D trajectory stream. Our system and method are the first in reconstructing full body motion of more than five people engaged in social interactions without using markers. We also empirically demonstrate the impact of the number of views in achieving this goal.

INCREMENTAL REAL-TIME BUNDLE ADJUSTMENT FOR MULTI-CAMERA SYSTEMS WITH POINTS AT INFINITY

Abstract. This paper presents a concept and first experiments on a keyframe-based incremental bundle adjustment for real-time structure and motion estimation in an unknown scene. In order to avoid periodic batch steps, we use the software iSAM2 for sparse nonlinear incremental optimization, which is highly efficient through incremental variable reordering and fluid relinearization. We adapted the software to allow for (1) multi-view cameras by taking the rigid transformation between the cameras into account, (2) omnidirectional cameras as it can handle arbitrary bundles of rays and (3) scene points at infinity, which improve the estimation of the camera orientation as points at the horizon can be observed over long periods of time. The real-time bundle adjustment refers to sets of keyframes, consisting of frames, one per camera, taken in a synchronized way, that are initiated if a minimal geometric distance to the last keyframe set is exceeded. It uses interest points in the keyframes as observations, which are tracked in the synchronized video streams of the individual cameras and matched across the cameras, if possible. First experiments show the potential of the incremental bundle adjustment w.r.t. time requirements. Our experiments are based on a multi-camera system with four fisheye cameras, which are mounted on a UAV as two stereo pairs, one looking ahead and one looking backwards, providing a large field of view.

An Effective Hybrid Depth-Generation Algorithm for 2D-to-3D Conversion in 3D Displays

In recent years, 3D display technology has been receiving increasingly more attention. The most intuitive 3D method is to use two temporally synchronized video streams for the left and right eyes, respectively. However, traditional 2D video contents are captured by one camera, and in order to synthesize the left and right views as the behavior of two cameras, depth map information is required. In this paper, we propose a hybrid algorithm for 2D-to-3D conversion in 3D displays; it is a good way to solve the problem of traditional 2D video contents which need to generate 3D effects in 3D displays. We choose three depth cues for depth estimation: motion information, linear perspective, and texture characteristics. Moreover, we adopt a bilateral filter for depth map smoothing and noise removal. From the experimental results, execution time can be reduced by 25%-35% and the depth perception score is between 75 and 85. Thus, the human eye cannot sense the noticeable differences from the final 3D rendering. Furthermore, it is very suitable to apply our proposed hybrid algorithm to 2D-to-3D conversion in 3D displays.

Multiview human pose estimation with unconstrained motions

We present a system for human pose estimation by using a single frame and without making assumptions on temporal coherence. The system uses 3D voxel data reconstructed from multiple synchronized video streams as input, and computes, for each frame, a skeleton model which best fits the body pose. This system adopts a hierarchical approach where the head and torso locations are found first based on template fitting with their specific shapes and dimensions. It is followed by a limb detection procedure that estimates the pose parameters of four limbs. However, a problem generally faced with skeleton models is the means to find adequate measurements to fit the model. In this paper, voxel data, together with two novel local shape features, are used for this purpose. Experiments show that this system is robust to several perturbations associated with the input data, such as voxel reconstruction errors and complex poses with self-contact, and also allows unconstrained motions, such as fast or unpredictable movements.

Markerless reconstruction and synthesis of dynamic facial expressions

In this paper we combine methods from the field of computer vision with surface editing techniques to generate animated faces, which are all in full correspondence to each other. The inputs for our system are synchronized video streams from multiple cameras. The system produces a sequence of triangle meshes with fixed connectivity, representing the dynamics of the captured face. By carefully taking all requirements and characteristics into account we decided for the proposed system design: We deform an initial face template using movements estimated from the video streams. To increase the robustness of the reconstruction, we use a morphable model as a shape prior to initialize a surfel fitting technique which is able to precisely capture face shapes not included in the morphable model. In the deformation stage, we use a 2D mesh based tracking approach to establish correspondences over time. We then reconstruct positions in 3D using the same surfel fitting technique, and finally use the reconstructed points to robustly deform the initially reconstructed face. We demonstrate the applicability of the tracked face template for automatic modeling and show how to use deformation transfer to attenuate expressions, blend expressions or how to build a statistical model, similar to a morphable model, on the dynamic movements.

Multicamera People Tracking with a Probabilistic Occupancy Map

Given two to four synchronized video streams taken at eye level and from different angles, we show that we can effectively combine a generative model with dynamic programming to accurately follow up to six individuals across thousands of frames in spite of significant occlusions and lighting changes. In addition, we also derive metrically accurate trajectories for each one of them. Our contribution is twofold. First, we demonstrate that our generative model can effectively handle occlusions in each time frame independently, even when the only data available comes from the output of a simple background subtraction algorithm and when the number of individuals is unknown a priori. Second, we show that multi-person tracking can be reliably achieved by processing individual trajectories separately over long sequences, provided that a reasonable heuristic is used to rank these individuals and avoid confusing them with one another.

High-Quality Reconstruction from Multiview Video Streams

Three-dimensional (3-D) video processing is currently an active area of research that attracts scientists from many disciplines, including computer graphics, computer vision, electrical engineering, and video processing. They join their expertise to attack the very hard problem of reconstructing dynamic representations of real- world scenes from a sparse set of synchronized video streams. To put this idea into practice, a variety of challenging engineering and algorithmic problems have to be efficiently solved, ranging from acquisition, over reconstruction in itself, to realistic rendering. This article is a tutorial style review of methods from the literature aiming at reconstruction of 3-D humans as well as of a variety of model-based approaches that we developed to reconstruct, render, and encode free-viewpoint videos of human actors. We will show that the commitment to an a priori shape representation of a person in the real world allows us to solve many of the previously described reconstruction problems in an efficient way.

Defocus video matting

Video matting is the process of pulling a high-quality alpha matte and foreground from a video sequence. Current techniques require either a known background (e.g., a blue screen) or extensive user interaction (e.g., to specify known foreground and background elements). The matting problem is generally under-constrained, since not enough information has been collected at capture time. We propose a novel, fully autonomous method for pulling a matte using multiple synchronized video streams that share a point of view but differ in their plane of focus. The solution is obtained by directly minimizing the error in filter-based image formation equations, which are over-constrained by our rich data stream. Our system solves the fully dynamic video matting problem without user assistance: both the foreground and background may be high frequency and have dynamic content, the foreground may resemble the background, and the scene is lit by natural (as opposed to polarized or collimated) illumination.

High-quality video view interpolation using a layered representation

The ability to interactively control viewpoint while watching a video is an exciting application of image-based rendering. The goal of our work is to render dynamic scenes with interactive viewpoint control using a relatively small number of video cameras. In this paper, we show how high-quality video-based rendering of dynamic scenes can be accomplished using multiple synchronized video streams combined with novel image-based modeling and rendering algorithms. Once these video streams have been processed, we can synthesize any intermediate view between cameras at any time, with the potential for space-time manipulation.In our approach, we first use a novel color segmentation-based stereo algorithm to generate high-quality photoconsistent correspondences across all camera views. Mattes for areas near depth discontinuities are then automatically extracted to reduce artifacts during view synthesis. Finally, a novel temporal two-layer compressed representation that handles matting is developed for rendering at interactive rates.

Human Body Model Acquisition and Tracking Using Voxel Data

We present an integrated system for automatic acquisition of the human body model and motion tracking using input from multiple synchronized video streams. The video frames are segmented and the 3D voxel reconstructions of the human body shape in each frame are computed from the foreground silhouettes. These reconstructions are then used as input to the model acquisition and tracking algorithms. The human body model consists of ellipsoids and cylinders and is described using the twists framework resulting in a non-redundant set of model parameters. Model acquisition starts with a simple body part localization procedure based on template fitting and growing, which uses prior knowledge of average body part shapes and dimensions. The initial model is then refined using a Bayesian network that imposes human body proportions onto the body part size estimates. The tracker is an extended Kalman filter that estimates model parameters based on the measurements made on the labeled voxel data. A voxel labeling procedure that handles large frame-to-frame displacements was designed resulting in very robust tracking performance. Extensive evaluation shows that the system performs very reliably on sequences that include different types of motion such as walking, sitting, dancing, running and jumping and people of very different body sizes, from a nine year old girl to a tall adult male.