Low Frame Rate Video Research Articles

Arbitrary Timestep Video Frame Interpolation with Time-Dependent Decoding

Given an observed low frame rate video, video frame interpolation (VFI) aims to generate a high frame rate video, which has smooth video frames with higher frames per second (FPS). Most existing VFI methods often focus on generating one frame at a specific timestep, e.g., 0.5, between every two frames, thus lacking the flexibility to increase the video’s FPS by an arbitrary scale, e.g., 3. To better address this issue, in this paper, we propose an arbitrary timestep video frame interpolation (ATVFI) network with time-dependent decoding. Generally, the proposed ATVFI is an encoder–decoder architecture, where the interpolation timestep is an extra input added to the decoder network; this enables ATVFI to interpolate frames at arbitrary timesteps between input frames and to increase the video’s FPS at any given scale. Moreover, we propose a data augmentation method, i.e., multi-width window sampling, where video frames can be split into training samples with multiple window widths, to better leverage training frames for arbitrary timestep interpolation. Extensive experiments were conducted to demonstrate the superiority of our model over existing baseline models on several testing datasets. Specifically, our model trained on the GoPro training set achieved 32.50 on the PSNR metric on the commonly used Vimeo90k testing set.

Retracted: Research on Low Frame Rate Video Compression Algorithm in the Context of New Media

Retracted: Research on Low Frame Rate Video Compression Algorithm in the Context of New Media

SuperFast: 200× Video Frame Interpolation via Event Camera.

Traditional frame-based video frame interpolation (VFI) methods rely on the linear motion assumption and brightness invariance assumption, which may lead to fatal errors confronting the scenarios with high-speed motions. To tackle the above challenge, inspired by the advantages of event cameras on asynchronously recording brightness changes at each pixel, we propose a Fast-Slow joint synthesis framework for event-enhanced high-speed video frame interpolation, named SuperFast, in this paper, which can generate high frame rate (5000 FPS, 200× faster) video from the input low frame rate (25 FPS) video and the corresponding event stream. In our framework, the task is divided into two sub-tasks, i.e., video frame interpolation for the contents with and without high-speed motions, which are tackled by two corresponding branches, i.e., the fast synthesis pathway and the slow synthesis pathway. The fast synthesis pathway leverages a spiking neural network to encode the input event stream, and combines boundary frames to generate intermediate results through synthesis and refinement, targeting on contents with high-speed motions. The slow synthesis pathway stacks the two input boundary frames and the event stream to synthesize intermediate results, focusing on relatively slow-motion contents. Finally, a fusion module with a comparison loss is utilized to generate the final video frame interpolation results. We also build a hybrid visual acquisition system containing an event camera and a high frame rate camera, and collect the first 5000 FPS High-Speed Event-enhanced Video frame Interpolation (THU[Formula: see text]) dataset. To evaluate the performance of our proposed framework, we have conducted experiments on our THU[Formula: see text] dataset and the existing HS-ERGB dataset. Experimental results demonstrate that our proposed framework can achieve state-of-the-art 200× video frame interpolation performance under high-speed motion scenarios.

Turbulence video simulation using the spatiotemporal cross correlation of distortion fields.

Previous simulations of atmospheric turbulence in videos are computationally complex. The purpose of this study is to develop an efficient algorithm for simulating spatiotemporal videos affected by atmospheric turbulence, given a static image. We extend an existing method for the simulation of atmospheric turbulence in a single image by incorporating turbulence properties in the time domain and the blurring effect. We accomplish this through analysis of the correlation between turbulence image distortions in time and in space. The significance of this method is the ease with which it will be possible to produce a simulation, given properties of the turbulence (including turbulence strength, object distance, and height). We apply the simulation to low and high frame rate videos, and we show that the spatiotemporal cross correlation of the distortion fields in the simulated video matches the physical spatiotemporal cross correlation function. Such a simulation can be useful when developing algorithms that apply to videos degraded by atmospheric turbulence and require a large amount of imaging data for training.

An extended TLD tracking algorithm using co-training learning for low frame rate videos

An extended TLD tracking algorithm using co-training learning for low frame rate videos

Video Frame Interpolation With Learnable Uncertainty and Decomposition

Video frame interpolation can flexibly increase the temporal resolution of low frame-rate videos by generating the missing intermediate frames at any time. Existing methods generally train a renderer to predict rgb frames based on estimated clues. It tends to generate blurry outputs with unpleasing artifacts due to false clues and the neural network's bias in favor of low-frequency information. To address this problem, we propose a novel two-stage supervised framework. The inaccuracy of clues is modeled as uncertainty which can be estimated by training implicitly with a parameterized loss function in stage one. The bias is alleviated in stage two by regressing a lossless decomposition of frames instead of the raw rgbs. The decomposition can be achieved by several invertible cross-coupling layers, motivating the network to synthesize high-frequency details. Moreover, the proposed framework is equipped with a time-varying neural network that is adaptive to the timestamp of any intermediate frame, bringing benefits to multiple-frame interpolation. Both qualitative and quantitative experiments demonstrate the superiority of our proposed approach.

An optimized complex motion prediction approach based on a video synopsis

PurposeThe space-time variants algorithm will not give good results in practical scenarios; when no tubes increase, these techniques will not give the results. It is challenging to reduce the energy of the output synopsis videos. In this paper, a new optimized technique has been implemented that models and covers every frame in the output video.Design/methodology/approachIn the video synopsis, condensing a video to produce a low frame rate (FR) video using their spatial and temporal coefficients is vital in complex environments. Maintaining a database is also feasible and consumes space. In recent years, many algorithms were proposed.FindingsThe main advantage of this proposed technique is that the output frames are selected by the user definitions and stored in low-intensity communication systems and also it gives tremendous support to the user to select desired tubes and thereby stops the criterion in the output video, which can be further suitable for the user's knowledge and creates nonoverlapping tube-oriented synopsis that can provide excellent visual experience.Research limitations/implicationsIn this research paper, four test videos are utilized with complex environments (high-density objects) and show that the proposed technique gives better results when compared to other existing techniques.Originality/valueThe proposed method provides a unique technique in video synopsis for compressing the data without loss.

Research on Low Frame Rate Video Compression Algorithm in the Context of New Media

When using the current method to compress the low frame rate video animation video, there is no frame rate compensation for the video image, which cannot eliminate the artifacts generated in the compression process, resulting in low definition, poor quality, and low compression efficiency of the compressed low frame rate video animation video. In the context of new media, the linear function model is introduced to study the frame rate video animation video compression algorithm. In this paper, an adaptive detachable convolutional network is used to estimate the offset of low frame rate video animation using local convolution. According to the estimation results, the video frames are compensated to eliminate the artifacts of low frame rate video animation. After the frame rate compensation, the low frame rate video animation video is divided into blocks, the CS value of the image block is measured, the linear estimation of the image block is carried out by using the linear function model, and the compression of the low frame rate video animation video is completed according to the best linear estimation result. The experimental results show that the low frame rate video and animation video compressed by the proposed algorithm have high definition, high compression quality under different compression ratios, and high compression efficiency under different compression ratios.

Three-Dimensional Stereo Vision Tracking of Multiple Free-Swimming Fish for Low Frame Rate Video

Three-dimensional multiple fish tracking has gained significant research interest in quantifying fish behavior. However, most tracking techniques use a high frame rate, which is currently not viable for real-time tracking applications. This study discusses multiple fish-tracking techniques using low-frame-rate sampling of stereo video clips. The fish were tagged and tracked based on the absolute error of the predicted indices using past and present fish centroid locations and a deterministic frame index. In the predictor sub-system, linear regression and machine learning algorithms intended for nonlinear systems, such as the adaptive neuro-fuzzy inference system (ANFIS), symbolic regression, and Gaussian process regression (GPR), were investigated. The results showed that, in the context of tagging and tracking accuracy, the symbolic regression attained the best performance, followed by the GPR, that is, 74% to 100% and 81% to 91%, respectively. Considering the computation time, symbolic regression resulted in the highest computing lag of approximately 946 ms per iteration, whereas GPR achieved the lowest computing time of 39 ms.

AMTSet: a benchmark for abrupt motion tracking

Since the OTB100 benchmark dataset is released, it has been widely used in a large number of researches on object tracking for performance evaluation. However, the existing datasets are insufficient to evaluate trackers in handling different challenging factors. In this paper, we present the first dataset and benchmark for tracking objects with abrupt motion (AMTSet). The dataset consists of 50 videos of special scenes from our real life, such as camera switching, sudden dynamic change, low frame rate video, etc., which are quite challenging in object tracking. Boundary boxes over 10K frames are marked manually, and all of them are manually labelled for common attributes of object tracking, such as scale variation, illumination variation, occlusion, motion blur, etc. We benchmark the dataset on 36 representative trackers and rank them according to the tracking conditions and results. Furthermore, we propose an evaluation measure for object tracking to better highlight the performances of the trackers against abrupt motion. Our goal is to supplement the existing baseline datasets and provide researchers with more perfect baseline data in order to better evaluate the performance of different trackers.

Video frame interpolation using deep cascaded network structure

Video frame interpolation is a technology that generates high frame rate videos from low frame rate videos by using the correlation between consecutive frames. Presently, convolutional neural networks (CNN) exhibit outstanding performance in image processing and computer vision. Many variant methods of CNN have been proposed for video frame interpolation by estimating either dense motion flows or kernels for moving objects. However, most methods focus on estimating accurate motion. In this study, we exhaustively analyze the advantages of both motion estimation schemes and propose a cascaded system to maximize the advantages of both the schemes. The proposed cascaded network consists of three autoencoder networks, that process the initial frame interpolation and its refinement. The quantitative and qualitative evaluations demonstrate that the proposed cascaded structure exhibits a promising performance compared to currently existing state-of-the-art-methods.

Forgery detection of motion compensation interpolated frames based on discontinuity of optical flow

Motion Compensated Frame Interpolation (MCFI), a frame-based video operation to increase the motion continuity of low frame rate video, can be adopted by falsifiers for forging high bitrate video or splicing videos with different frame-rates. For existing MCFI detectors, their performance are degraded by stronger video compression, and noise. To deal with this problem, we propose a blind forensics method to detect the adopted MCFI operation. After investigating the synthetic process of interpolated frames, we discover that motion regions of interpolated frames exist some local slight artifacts, causing the optical flow based inter-frame discontinuity. To capture this irregularities introduced by various MCFI techniques, compact features are designed, which are calculated as Temporal Frame Difference-weighted histogram of Local Binary Pattern computed on Optical Flow field (TFD-OFLBP). Meanwhile, Local Inter-block and Edge-block difference Features (LIEF) are further proposed to detect interpolation frames with stable content. Besides, a set of forensics tools are adopted to eliminate the side effects of possible interferences of the scenes change, sudden lighting change, focus vibration, and some original frames with inherent local artifacts. Experimental results on four representative MCFI software and techniques show that the proposed approach outperforms existing MCFI detectors and also with robustness to compression, and noise.

An extended KCF tracking algorithm based on TLD structure in low frame rate videos

An extended KCF tracking algorithm based on TLD structure in low frame rate videos

End‐to‐end learning interpolation for object tracking in low frame‐rate video

In many scenarios, where videos are transmitted through bandwidth-limited channels for subsequent semantic analytics, the choice of frame rates has to balance between bandwidth constraints and analytics performance. Faced with this practical challenge, this study focuses on enhancing object tracking at low frame rates and proposes a learning Interpolation for tracking framework. This framework embeds an implicit video frame interpolation sub-network, which is concatenated and jointly trained with another object tracking sub-network. Once a low frame-rate video is an input, it is first mapped into a high frame-rate latent video, based on which the tracker is learned. Novel strategies and loss functions are derived to ensure the effective end-to-end optimisation of the authors’ network. On several challenging benchmarks and settings, their method achieves a highly competitive tradeoff between frame rate and tracking accuracy. As is known, the implications of interpolation on semantic video analytics and tracking remain unexplored, and the authors expect their method to find many applications in mobile embedded vision, Internet of Things and edge computing.

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition.

This paper presents a dual camera system for high spatiotemporal resolution (HSTR) video acquisition, where one camera shoots a video with high spatial resolution and low frame rate (HSR-LFR) and another one captures a low spatial resolution and high frame rate (LSR-HFR) video. Our main goal is to combine videos from LSR-HFR and HSR-LFR cameras to create an HSTR video. We propose an end-to-end learning framework, AWnet, mainly consisting of a FlowNet and a FusionNet that learn an adaptive weighting function in pixel domain to combine inputs in a frame recurrent fashion. To improve the reconstruction quality for cameras used in reality, we also introduce noise regularization under the same framework. Our method has demonstrated noticeable performance gains in terms of both objective PSNR measurement in simulation with different publicly available video and light-field datasets and subjective evaluation with real data captured by dual iPhone 7 and Grasshopper3 cameras. Ablation studies are further conducted to investigate and explore various aspects, such as reference structure, camera parallax, exposure time, etc) of our system to fully understand its capability for potential applications.

Detection of motion-compensated frame-rate up-conversion via optical flow-based prediction residue

To increase the temporal continuity of low frame-rate videos, Motion-Compensated Frame-Rate Up-Conversion (MC-FRUC), which is a special frame based video editing manipulation, can be employed to synthesize new intermediate frame between two successive frames. However, MC-FRUC can also be utilized to fake high frame-rate videos causing the wastes of storage space or users misleading. Although existing MC-FRUC detectors can obtain desirable performance for uncompressed videos or compressed videos with high perceptual quality, theirs performance degrades when test videos are encoded with higher quantization parameters. To deal with this problem, this paper presents a new optical flow-based detection method. Our method is divided into four steps. First, the forward or backward optical flow map is estimated by optical flow algorithm between the candidate frame, and its previous frame or following frame, respectively. Second, the prediction residue is calculated on the basis of these estimated optical flow maps to eliminate the side effect of texture information in the non-motion regions and the motion regions. Third, Markov statistic features are extracted. Finally, these features are fed into the ensemble classifier for identification of interpolated frames. We test different types of MC-FRUC methods under higher quantization parameters, and the results have shown the effectiveness of the proposed method compared with the existing relative ones.

Robust Localization of Interpolated Frames by Motion-Compensated Frame Interpolation Based on an Artifact Indicated Map and Tchebichef Moments

Motion-compensated frame interpolation (MCFI), a frame-interpolation technique to increase the motion continuity of low frame-rate video, can be utilized by counterfeiters for faking high bitrate video or splicing videos with different frame rates. For existing MCFI detectors, their performances are degraded under real-world scenarios such as H.264/AVC compression, noise, or blur. To address this issue, a robust MCFI detector is proposed to locate interpolated frames. By analyzing the distribution of residual energies within interpolated frames, we observe that there exist strong correlations between artifact regions and high residual energies. Thus, an artifact indicated map is introduced to select candidate artifact regions. Then, Tchebichef moments (TMs) are exploited to characterize the blurring effects or deformed structures among these regions. Specifically, the mean value of absolute high-order TMs of selected regions is used to model these temporal inconsistencies. Finally, a sliding window is adopted to locate interpolated frames, which are further refined by three post-processing operations. Chrominance information is also integrated with luminance information for robust identification of interpolated frames. Extensive experimental results show that compared with the state-of-the-art MCFI detectors, the proposed approach is more robust for compressed videos under various real-world scenarios.

Validation of a videogrammetry technique for analysing American football helmet kinematics

ABSTRACT Professional American football games are recorded in digital video with multiple cameras, often at high resolution and high frame rates. The purpose of this study was to evaluate the accuracy of a videogrammetry technique to calculate translational and rotational helmet velocity before, during and after a helmet impact. In total, 10 football impacts were staged in a National Football League (NFL) stadium by propelling helmeted 50th percentile male crash test dummies into each other or the ground at speeds and orientations representative of concussive impacts for NFL players. The tests were recorded by experienced sports film crews to obtain video coverage and quality typically available for NFL games. A videogrammetry procedure was used to track the position and rotation of the helmet throughout the relevant time interval of the head impact. Compared with rigidly mounted retroreflective marker three dimensional (3-D) motion tracking that was concurrently collected in the experiments, videogrammetry accurately calculated changes in translational and rotational velocity of the helmet using high frame rate (two cameras at 240 Hz) video (7% and 15% error, respectively). Low frame rate (2 cameras at 60 Hz) video was adequate for calculating pre-impact translational velocity but not for calculating the translational or rotational velocity change of the helmet during impact.

Parallel implementations of frame rate up-conversion algorithm using OpenCL on heterogeneous computing devices

As a video post-processing technology, frame rate up-conversion (FRUC) converts a low frame rate video into a higher one by inserting intermediate frames between adjacent original frames. Because computing consumption grows rapidly with the increase of video resolution and frame rate, accelerating FRUC by parallel computing may serve as an appropriate method. In this paper, an effective parallel FRUC algorithm is proposed, which consists mainly of two parts: parallel motion estimation algorithm (Three-dimensional Recursive Search algorithm, 3DRS algorithm) and parallel motion compensation algorithm. We design macro-block-level parallelism and candidate motion vector level parallelism strategies based on different granularity in the motion estimation module, and pixel-level parallelism in the motion compensation module. The proposed parallel FRUC algorithm has been tested on different hardware platforms. The results show that the method achieves significant speedups of up to 96× for 1920 × 1080 video and 254× for 3840 × 2160 video when compared with sequential implementation on CPU. Moreover, the OpenCL program of the parallel FRUC algorithm shows good portability on various GPU platforms.

Detection of motion compensated frame interpolation via motion-aligned temporal difference

Motion compensated frame interpolation (MCFI) is a special frame based video manipulation, which increases the temporal continuity of low frame rate videos by synthesizing new frames between successive frames. MCFI might also be used to counterfeit high frame-rate videos, which mislead users’ attraction and waste storage spaces in video-sharing websites. Existing MCFI detectors are designed to judge the absence or presence of MCFI forgery in a controllable environment of known MCFI techniques. Practical detector should consider the possibility of unknown MCFI techniques. We are motivated to propose a robust MCFI detector for more practical scenarios. Considering the effects of non-motion regions in candidate frame, the statistical moments are firstly extracted from motion-aligned frame differences (MAFD). Then, the one-class support vector machine (SVM), following a training stage capturing the properties of original frames, is exploited to judge whether the candidate frame is interpolated by MCFI or not. Finally, a special interpolated frame detection (SIFD) is designed to pick out interpolated frames, which are synthesized from two consecutive reference frames with no motion vectors (MVs) or less MVs. A series of experiments evaluated on four representative MCFI techniques have shown promising results.