Arbitrary Video Sequences Research Articles

Regenerating Arbitrary Video Sequences With Distillation Path-Finding.

If the video has long been mentioned as a widespread visualization form, the animation sequence in the video is mentioned as storytelling for people. Producing an animation requires intensive human labor from skilled professional artists to obtain plausible animation in both content and motion direction, incredibly for animations with complex content, multiple moving objects, and dense movement. This article presents an interactive framework to generate new sequences according to the users' preference on the starting frame. The critical contrast of our approach versus prior work and existing commercial applications is that novel sequences with arbitrary starting frame are produced by our system with a consistent degree in both content and motion direction. To achieve this effectively, we first learn the feature correlation on the frameset of the given video through a proposed network called RSFNet. Then, we develop a novel path-finding algorithm, SDPF, which formulates the knowledge of motion directions of the source video to estimate the smooth and plausible sequences. The extensive experiments show that our framework can produce new animations on the cartoon and natural scenes and advance prior works and commercial applications to enable users to obtain more predictable results.

Preview the Products on Display at Display Week 2023

Preview the Products on Display at Display Week 2023

PC-SD-VIO: A constant intensity semi-direct monocular visual-inertial odometry with online photometric calibration

The brightness constancy assumption is the cornerstone of direct or semi-direct visual odometry (VO) and visual simultaneous localization and mapping (SLAM). However, due to the existence of automatic exposure time, nonlinear camera response function, and vignetting, this assumption is difficult to hold in practical applications. Therefore, the corresponding algorithm performs poorly on arbitrary video sequences and uncalibrated cameras. Hence, we propose a novel constant intensity semi-direct visual-inertial odometry (VIO) integrated with online photometric calibration, which combines the exactness of the feature-based method and the quickness of the direct method. We combine gain-adaptive direct image alignment and gain-adaptive Kanade–Lucas–Tomasi (KLT) optical flow tracking to complete the feature matching and use it as the input for the back-end optimization and online photometric calibration. Our photometric calibration module can complete the estimation of all photometric parameters without any prior knowledge and cooperate with the front-end to complete the real-time photometric calibration of the latest frame. Experiments on the TUM Mono VO dataset, EuRoC dataset, and real environments prove that the algorithm can reliably calibrate the photometric parameters of an arbitrary video sequence. The semi-direct VIO algorithm integrated with the photometric calibration algorithm achieves a good balance between speed, accuracy, and robustness.

Enhanced 3D Human Pose Estimation from Videos by Using Attention-Based Neural Network with Dilated Convolutions

The attention mechanism provides a sequential prediction framework for learning spatial models with enhanced implicit temporal consistency. In this work, we show a systematic design (from 2D to 3D) for how conventional networks and other forms of constraints can be incorporated into the attention framework for learning long-range dependencies for the task of pose estimation. The contribution of this paper is to provide a systematic approach for designing and training of attention-based models for the end-to-end pose estimation, with the flexibility and scalability of arbitrary video sequences as input. We achieve this by adapting temporal receptive field via a multi-scale structure of dilated convolutions. Besides, the proposed architecture can be easily adapted to a causal model enabling real-time performance. Any off-the-shelf 2D pose estimation systems, e.g. Our method achieves the state-of-the-art performance and outperforms existing methods by reducing the mean per joint position error to 33.4mm on Human 3.6M dataset. Our code is available at https://github.com/lrxjason/Attention3DHumanPose

OneShotDA: Online Multi-Object Tracker With One-Shot-Learning-Based Data Association

Tracking multiple objects in a video sequence can be accomplished by identifying the objects appearing in the sequence and distinguishing between them. Therefore, many recent multi-object tracking (MOT) methods have utilized re-identification and distance metric learning to distinguish between objects by computing the similarity/dissimilarity scores. However, it is difficult to generalize such approaches for arbitrary video sequences, because some important information, such as the number of objects (classes) in a video, is not known in advance. Therefore, in this study, we applied a one-shot learning framework to the MOT problem. Our algorithm tracks objects by classifying newly observed objects into existing tracks, irrespective of the number of objects appearing in a video frame. The proposed method, called OneShotDA , exploits the one-shot learning framework based on an attention mechanism. Our neural network learns to classify unseen data samples using labels from a support set. Once the network has been trained, it predicts correct labels for newly received detection results based on the set of existing tracks. To analyze the effectiveness of our method, it was tested on the MOTchallenge benchmark datasets (MOT16 and MOT17 datasets). The results reveal that the performance of the proposed method was comparable with those of current state-of-the-art methods. In particular, it is noteworthy that the proposed method ranked first among the online trackers on the MOT17 benchmark.

Online Photometric Calibration of Auto Exposure Video for Realtime Visual Odometry and SLAM

Recent direct visual odometry and SLAM algorithms have demonstrated impressive levels of precision. However, they require a photometric camera calibration in order to achieve competitive results. Hence, the respective algorithm cannot be directly applied to an off-the-shelf-camera or to a video sequence acquired with an unknown camera. In this letter, we propose a method for online photometric calibration that enables to process auto exposure videos with visual odometry precisions that are on par with those of photometrically calibrated videos. Our algorithm recovers the exposure times of consecutive frames, the camera response function, and the attenuation factors of the sensor irradiance due to vignetting. Gain robust Kanade-Lucas-Tomasi (KLT) feature tracks are used to obtain scene point correspondences as input to a nonlinear optimization framework. We show that our approach can reliably calibrate arbitrary video sequences by evaluating it on datasets for which full photometric ground truth is available. We further show that our calibration can improve the performance of a state-of-the-art direct visual odometry method that works solely on pixel intensities, calibrating for photometric parameters in an online fashion in realtime.

Camera–projector matching using unstructured video

This paper presents a novel approach for matching 2-D points between a video projector and a digital camera. Our method is motivated by camera–projector applications for which the projected image needs to be warped to prevent geometric distortion. Since the warping process often needs geometric information on the 3-D scene obtained from a triangulation, we propose a technique for matching points in the projector to points in the camera based on arbitrary video sequences. The novelty of our method lies in the fact that it does not require the use of pre-designed structured light patterns as is usually the case. The backbone of our application lies in a function that matches activity patterns instead of colors. This makes our method robust to pose, severe photometric and geometric distortions. It also does not require calibration of the color response curve of the camera–projector system. We present quantitative and qualitative results with synthetic and real-life examples, and compare the proposed method with the scale invariant feature transform (SIFT) method and with a state-of-the-art structured light technique. We show that our method performs almost as well as structured light methods and significantly outperforms SIFT when the contrast of the video captured by the camera is degraded.

ETFRC: Enhanced TFRC for Media Traffic

Due to the remarkable increase in media traffic over Internet, its congestion state is expected to worsen. TCP-friendly rate control protocol TFRC is one of the most promising congestion control techniques developed so far. TFRC has been thoroughly tested in terms of being TCP-friendly, responsive, and fair. Yet, its impact on the visual quality and the peak signal-tonoise ratio PSNR of the media traffic traversing Internet is still questionable. In this paper we aimed to point out the enhancement required for TFRC that enables producing the maximum PSNR value for Internet media traffic. Firstly, we suspected the default value of Nfb that represents the frequency of feedback messages sent by TFRC receiver to its sender every round-trip time RTT to be the optimum for our goal. Secondly, we managed to modify the mechanism of TFRC by changing this default value of Nfb over a range of values aiming to reach the required optimum. Thirdly, we investigated the effect of such variation over a simulated network environment to study its effect on the resulting PSNR for a number of arbitrary video sequences. Finally, our simulations results showed that running TFRC with Nfb=4 led to reaching the maximum PSNR values among all the examined values of Nfb including its default value. We hereby suggest using an enhanced TFRC that we abbreviated as ETFRC which has the Nfb value set to four as a replacement for the traditional TFRC to enable reaching higher PSNR for media traffic over Internet. Kerwords—TFRC, Congestion Control, PSNR

Integrating Motion, Illumination, and Structure in Video Sequences with Applications in Illumination-Invariant Tracking

In this paper, we present a theory for combining the effects of motion, illumination, 3D structure, albedo, and camera parameters in a sequence of images obtained by a perspective camera. We show that the set of all Lambertian reflectance functions of a moving object, at any position, illuminated by arbitrarily distant light sources, lies "close" to a bilinear subspace consisting of nine illumination variables and six motion variables. This result implies that, given an arbitrary video sequence, it is possible to recover the 3D structure, motion, and illumination conditions simultaneously using the bilinear subspace formulation. The derivation builds upon existing work on linear subspace representations of reflectance by generalizing it to moving objects. Lighting can change slowly or suddenly, locally or globally, and can originate from a combination of point and extended sources. We experimentally compare the results of our theory with ground truth data and also provide results on real data by using video sequences of a 3D face and the entire human body with various combinations of motion and illumination directions. We also show results of our theory in estimating 3D motion and illumination model parameters from a video sequence.

Using the low-resolution properties of correlated images to improve the computational efficiency of eigenspace decomposition

Eigendecomposition is a common technique that is performed on sets of correlated images in a number of computer vision and robotics applications. Unfortunately, the computation of an eigendecomposition can become prohibitively expensive when dealing with very high-resolution images. While reducing the resolution of the images will reduce the computational expense, it is not known a priori how this will affect the quality of the resulting eigendecomposition. The work presented here provides an analysis of how different resolution reduction techniques affect the eigendecomposition. A computationally efficient algorithm for calculating the eigendecomposition based on this analysis is proposed. Examples show that this algorithm performs well on arbitrary video sequences.

Fast eigenspace decomposition of correlated images

We present a computationally efficient algorithm for the eigenspace decomposition of correlated images. Our approach is motivated by the fact that for a planar rotation of a two-dimensional (2-D) image, analytical expressions can be given for the eigendecomposition, based on the theory of circulant matrices. These analytical expressions turn out to be good first approximations of the eigendecomposition, even for three-dimensional (3-D) objects rotated about a single axis. In addition, the theory of circulant matrices yields good approximations to the eigendecomposition for images that result when objects are translated and scaled. We use these observations to automatically determine the dimension of the subspace required to represent an image with a guaranteed user-specified accuracy, as well as to quickly compute a basis for the subspace. Examples show that the algorithm performs very well on a number of test cases ranging from images of 3-D objects rotated about a single axis to arbitrary video sequences.

An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis

Key frames and previews are two forms of a video abstract, widely used for various applications in video browsing and retrieval systems. We propose in this paper a novel method for generating these two abstract forms for an arbitrary video sequence. The underlying principle of the proposed method is the removal of the visual-content redundancy among video frames. This is done by first applying multiple partitional clustering to all frames of a video sequence and then selecting the most suitable clustering option(s) using an unsupervised procedure for cluster-validity analysis. In the last step, key frames are selected as centroids of obtained optimal clusters. Video shots, to which key frames belong, are concatenated to form the preview sequence.