Synthetic Video Sequences Research Articles

In-the-wild Material Appearance Editing using Perceptual Attributes

Intuitively editing the appearance of materials, just from a single image, is a challenging task given the complexity and ambiguity of the interactions between light and matter. This problem has been traditionally solved by estimating additional factors of the scene like geometry or illumination, thus solving an inverse rendering problem where the interaction of light and matter needs to be modelled. Instead, we present a single-image appearance editing framework that allows to intuitively modify the material appearance of an object by increasing or decreasing high-level perceptual attributes describing appearance (e.g., glossy or metallic). Our framework uses just an in-the-wild image as input, where geometry or illumination are not controlled.
 We rely on generative neural networks and, in particular, on Selective Transfer Generative Adversarial Networks (STGAN) that allow to preserve high-frequency details from the input image in the edited one. To train our framework we combine pairs of synthetic images, rendered with physically-based ray tracing algorithms, and their corresponding ratings of the high-level attributes, given by humans through crowd-sourced user studies. Last, although trained on synthetic images, we demonstrate the applicability of our method on synthetic video sequences; and real-world photographs downloaded from online catalogs and manually taken using our mobile phones.

Vision System Measuring the Position of an Aircraft in Relation to the Runway during Landing Approach.

This paper presents a vision system that measures the position of an aircraft relative to the runway (RWY) during a landing approach. It was assumed that all the information necessary for a correct approach was based entirely on an analysis of the image of the runway and its surroundings. It was assumed that the way the algorithm works, as well as possible, should imitate the pilot's perception of the runway. Taking into account the above and the fact that the infrastructure at each airport is different, it has been decided to use artificial neural networks with a dedicated learning process for any airport, based on the simulation environments. Such an action will enable the generation of a synthetic video sequence without the need for costly and time-consuming flights. The presented solution was tested in real flight conditions on an experimental aircraft, and the selected test results are presented in this article.

Real-Time Ship Segmentation in Maritime Surveillance Videos Using Automatically Annotated Synthetic Datasets.

This work proposes a new system capable of real-time ship instance segmentation during maritime surveillance missions by unmanned aerial vehicles using an onboard standard RGB camera. The implementation requires two stages: an instance segmentation network able to produce fast and reliable preliminary segmentation results and a post-processing 3D fully connected Conditional Random Field, which significantly improves segmentation results by exploring temporal correlations between nearby frames in video sequences. Moreover, due to the absence of maritime datasets consisting of properly labeled video sequences, we create a new dataset comprising synthetic video sequences of maritime surveillance scenarios (MarSyn). The main advantages of this approach are the possibility of generating a vast set of images and videos, being able to represent real-world scenarios without the necessity of deploying the real vehicle, and automatic labels, which eliminate human labeling errors. We train the system with the MarSyn dataset and with aerial footage from publicly available annotated maritime datasets to validate the proposed approach. We present some experimental results and compare them to other approaches, and we also illustrate the temporal stability provided by the second stage in missing frames and wrong segmentation scenarios.

Regularized kernel matrix decomposition for thermal video multi-object detection and tracking

This paper derives a novel algorithm for joint detection and tracking of multiple moving objects in thermal videos. The problem of determining multiple objects in a frame sequence is formulated as the task of factorizing a properly defined kernel covariance matrix into sparse factors. The support of these factors will point to the indices of the pixels that form each object. A coordinate descent approach is utilized to determine the sparse factors, and extract the object pixels. A centroid pixel is estimated for each object which is subsequently tracked via Kalman filtering. A novel interplay between the sparse kernel covariance factorization scheme along with Kalman filtering is proposed to enable joint object detection and tracking, while a divide and conquer strategy is put forth to reduce computational complexity and enable efficient tracking. Extensive numerical tests on both synthetic data and thermal video sequences demonstrate the effectiveness of the novel approach and superior tracking performance compared to existing alternatives.

Combinatorial Resampling Particle Filter: An Effective and Efficient Method for Articulated Object Tracking

Particle filter (PF) is a method dedicated to posterior density estimations using weighted samples whose elements are called particles. In particular, this approach can be applied to object tracking in video sequences in complex situations and, in this paper, we focus on articulated object tracking, i.e., objects that can be decomposed as a set of subparts. One of PF's crucial step is a resampling step in which particles are resampled to avoid degeneracy problems. In this paper, we propose to exploit mathematical properties of articulated objects to swap conditionally independent subparts of the particles in order to generate new particle sets. We then introduce a new resampling method called Combinatorial Resampling that resamples over the particle set resulting from all the "admissible" swappings, the so-called combinatorial set. In essence, combinatorial resampling (CR) is quite similar to the combination of a crossover operator and a usual resampling, but there exists a fundamental difference between CR and the use of crossover operators: we prove that CR is sound, i.e., in a Bayesian framework, it is guaranteed to represent without any bias the posterior densities of the states over time. By construction, the particle sets produced by CR better represent the density to estimate over the whole state space than the original set and, therefore, CR produces higher quality samples. Unfortunately, the combinatorial set is generally of an exponential size and, therefore, to be scalable, we show how it can be implicitly constructed and resampled from, thus resulting in both an efficient and effective resampling scheme. Finally, through experimentations both on challenging synthetic and real video sequences, we also show that our resampling method outperforms all classical resampling methods both in terms of the quality of its results and in terms of computation times.

Sub-sample swapping for sequential Monte Carlo approximation of high-dimensional densities in the context of complex object tracking

In this paper, we address the problem of complex object tracking using the particle filter framework, which essentially amounts to estimate high-dimensional distributions by a sequential Monte Carlo algorithm. For this purpose, we first exploit Dynamic Bayesian Networks to determine conditionally independent subspaces of the object’s state space, which allows us to independently perform the particle filter’s propagations and corrections over small spaces. Second, we propose a swapping process to transform the weighted particle set provided by the update step of the particle filter into a “new particle set” better focusing on high peaks of the posterior distribution. This new methodology, called Swapping-Based Partitioned Sampling, is proved to be mathematically sound and is successfully tested and validated on synthetic video sequences for single or multiple articulated object tracking.

Modeling dynamic swarms

This paper proposes the problem of modeling video sequences of dynamic swarms (DSs). We define a DS as a large layout of stochastically repetitive spatial configurations of dynamic objects (swarm elements) whose motions exhibit local spatiotemporal interdependency and stationarity, i.e., the motions are similar in any small spatiotemporal neighborhood. Examples of DS abound in nature, e.g., herds of animals and flocks of birds. To capture the local spatiotemporal properties of the DS, we present a probabilistic model that learns both the spatial layout of swarm elements (based on low-level image segmentation) and their joint dynamics that are modeled as linear transformations. To this end, a spatiotemporal neighborhood is associated with each swarm element, in which local stationarity is enforced both spatially and temporally. We assume that the prior on the swarm dynamics is distributed according to an MRF in both space and time. Embedding this model in a MAP framework, we iterate between learning the spatial layout of the swarm and its dynamics. We learn the swarm transformations using ICM, which iterates between estimating these transformations and updating their distribution in the spatiotemporal neighborhoods. We demonstrate the validity of our method by conducting experiments on real and synthetic video sequences. Real sequences of birds, geese, robot swarms, and pedestrians evaluate the applicability of our model to real world data.

3D video visualization employing wavelet multilevel decomposition

This study analyzed the implementation and performance of a framework that can be efficiently applied to three-dimensional (3D) video sequence visualization. The proposed algorithm is based on wavelets and wavelet atomic functions used in the computation of disparity maps. The proposed algorithm employs wavelet multilevel decomposition and 3D visualization via color anaglyphs synthesis. Simulations were run on synthetic images, synthetic video sequences, and real-life video sequences. Results shows that this novel approach performs better in depth and spatial perception tasks compared to existing methods, both in terms of objective criteria such as quantity of bad disparities and similarity structural index measure and the more subjective measure of human vision.

Performance of optical flow techniques for motion analysis of fluorescent point signals in confocal microscopy

Optical flow approaches calculate vector fields which determine the apparent velocities of objects in time-varying image sequences. They have been analyzed extensively in computer science using both natural and synthetic video sequences. In life sciences, there is an increasing need to extract kinetic information from temporal image sequences which reveals the interplay between form and function of microscopic biological structures. In this work, we test different variational optical flow techniques to quantify the displacements of biological objects in 2D fluorescent image sequences. The accuracy of the vector fields is tested for defined displacements of fluorescent point sources in synthetic image series which mimic protein traffic in neuronal dendrites, and for GABABR1 receptor subunits in dendrites of hippocampal neurons. Our results reveal that optical flow fields predict the movement of fluorescent point sources within an error of 3% for a maximum displacement of 160 nm. Displacement of agglomerated GABABR1 receptor subunits can be predicted correctly for maximum displacements of 640 nm. Based on these results, we introduce a criteria to derive the optimum parameter combinations for the calculation of the optical flow fields in experimental images. From these results, temporal sampling frequencies for image acquisition can be derived to guarantee correct motion estimation for biological objects.

Geometrical recovery of missing data for shape from motion

This paper studies the geometrical recovery of an incomplete observation matrix for converting existing 2D video sequences to 3D content. In situations when converting previously recorded monoscopic video to 3D, several entries of the observation matrix have not been observed and other entries have been perturbed by the influence of noise. In such cases, there is no simple solution for SVD factorization for shape from motion. In this paper, a new recovery algorithm is proposed for recovering missing feature points, by minimizing the influence of noise, using iteratively geometrical correlations between a 2D observation matrix and 3D shape. Results in practical situations demonstrated with synthetic and real video sequences verify the efficiency and flexibility of the proposed method.

Accurate Estimation of Missing Data under Noise Distribution

3D contents have been becoming one of attractive multimedia and reality. In computer vision, 2D-to-3D conversion techniques require estimating missing data from noisy observations. When there is no missing data in the observation matrix, the accurate solution of such problem is known to be given by singular value decomposition (SVD). In the case of converting already recorded monoscopic video contents to 3D, several entries of the matrix have not been observed. Therefore, the problem has no simple solution, so it is necessary to estimate missing data. In this paper, we propose an estimation algorithm of missing data with minimizing the influence of noise embedded when tracking feature points from partial observations. The proposed method is an iterative affine SVD factorization method which can estimate the model parameters, given an incomplete set of the observation matrix. The main idea of our algorithm is to estimate missing data accurately even under noise distribution by using geometrical correlations between 2D and 3D error space. This paper consists of three main phases: geometrical correlations for estimating missing data, estimation algorithm, and analyzing the results for video sequences. The accurate results in practical situations as demonstrated here with synthetic and real video sequences show the efficiency and flexibility of the proposed method.

Perception-motivated high dynamic range video encoding

Due to rapid technological progress in high dynamic range (HDR) video capture and display, the efficient storage and transmission of such data is crucial for the completeness of any HDR imaging pipeline. We propose a new approach for inter-frame encoding of HDR video, which is embedded in the well-established MPEG-4 video compression standard. The key component of our technique is luminance quantization that is optimized for the contrast threshold perception in the human visual system. The quantization scheme requires only 10--11 bits to encode 12 orders of magnitude of visible luminance range and does not lead to perceivable contouring artifacts. Besides video encoding, the proposed quantization provides perceptually-optimized luminance sampling for fast implementation of any global tone mapping operator using a lookup table. To improve the quality of synthetic video sequences, we introduce a coding scheme for discrete cosine transform (DCT) blocks with high contrast. We demonstrate the capabilities of HDR video in a player, which enables decoding, tone mapping, and applying post-processing effects in real-time. The tone mapping algorithm as well as its parameters can be changed interactively while the video is playing. We can simulate post-processing effects such as glare, night vision, and motion blur, which appear very realistic due to the usage of HDR data.

Creating Walk-Through Images from a Video Sequence of a Dynamic Scene

Tour into the picture (TIP), proposed by Horry et al. (Horry, Anjyo, & Arai, 1997, ACM SIGGRAPH '97 Conference Proceedings, 225–232) is a method for generating a sequence of walk-through images from a single reference image. By navigating a 3D scene model constructed from the image, TIP provides convincing 3D effects. This paper presents a comprehensive scheme for creating walk-through images from a video sequence by generalizing the idea of TIP. To address various problems in dealing with a video sequence rather than a single image, the proposed scheme is designed to have the following features: First, it incorporates a new modeling scheme based on a vanishing circle identified in the video, assuming that the input video contains a negligible amount of motion parallax effects and that dynamic objects move on a flat terrain. Second, we propose a novel scheme for automatic background detection from the video, based on 4-parameter motion model and statistical background color estimation. Third, to assist the extraction of static or dynamic foreground objects from video, we devised a semiautomatic boundary-segmentation scheme based on enhanced lane (Kang & Shin, 2002, Graphical Models, 64 (5), 282–303). The purpose of this work is to let users experience the feel of navigating into a video sequence with their own interpretation and imagination about a given scene. The proposed scheme covers various types of video films of dynamic scenes, such as sports coverage, cartoon animation, and movie films, in which objects are continuously changing their shapes and locations. It can also be used to produce a variety of synthetic video sequences by importing and merging dynamic foreign objects with the original video.

Optimizing motion-vector accuracy in block-based video coding

All motion-vectors are encoded with the same tired accuracy, typically 1/2-pixel accuracy, but the best motion-vector accuracies are not known. We present a theoretical framework to find the motion-vector accuracies that minimize the total encoding rate with this type of coder, for the classical case where all motion-vectors are encoded with the same accuracy, and for new cases where the accuracy is adapted on a frame-by-frame or block-by-block basis. To do this, we analytically model the effect of motion-vector accuracy and show that the energy in a block of the difference frame is approximately quadratic in the accuracy of the block's motion-vector. This energy-accuracy model is then used to obtain expressions for the total bit rate (motion rate plus difference frame rate) in terms of the blocks' motion accuracies and other key parameters. Minimizing these expressions leads to simple formulas that indicate how to choose the best motion-vector accuracies for this type of coder. These formulas also show that the motion accuracy must increase where more texture is present and decrease when there is much scene noise or when the level of compression is high. We implement several entropy and MPEG-like video coders based on our analysis and present experimental results on synthetic and real video sequences. These results suggest that our formulas are accurate and that significant bit rate savings can be achieved when our optimization procedures are used.

Encoding Video Narration as Text

This paper describes a solution to the problem of transmitting “talking head” video in real-time across low-bandwidth transmission media. Our solution is based on a video reconstruction system, which can generate realistic audio-visual narrations from arbitrary pieces of text. The system uses standard synthetic speech techniques to create an audio track and then produces a synchronized “talking head” using full-frame morphing of real-world facial images (key-frames). We discuss issues of original data capture, transmission protocols, synchronization, interpolation, and the modeling of various physical activities which are vital to maintaining the “plausibility” of the final synthetic video sequence.