Image Sequence Interpolation Research Articles

100 kHz CH2O imaging realized by lower speed planar laser-induced fluorescence and deep learning.

This paper reports an approach to interpolate planar laser-induced fluorescence (PLIF) images of CH2O between consecutive experimental data by means of computational imaging realized with convolutional neural network (CNN). Such a deep learning based method can achieve higher temporal resolution for 2D visualization of intermediate species in combustion based on high-speed experimental images. The capability of the model was tested for generating 100 kHz PLIF images by interpolating single and multiple PLIF frames into the sequences of experimental images of lower frequencies (50, 33, 25 and 20 kHz). Results show that the prediction indices, including intersection over union (IoU), peak signal to noise ratio (PSNR), structural similarity index (SSIM), and time averaged correlation coefficient at various axial positions could achieve acceptable accuracy. This work sheds light on the utilization of CNN-based models to achieve optical flow computation and image sequence interpolation, also providing an efficient off-line model as an alternative pathway to overcome the experimental challenges of the state-of-the-art ultra-high speed PLIF techniques, e.g., to further increase repetition rate and save data transfer time.

Image Sequence Interpolation Based on Optical Flow, Segmentation, and Optimal Control

When using motion fields to interpolate between two consecutive images in an image sequence, a major problem is to handle occlusions and disclusions properly. However, in most cases, one of both images contains the information that is either discluded or occluded; if the first image contains the information (i.e., the region will be occluded), forward interpolation shall be employed, while for information that is contained in the second image (i.e., the region will be discluded), one should use backward interpolation. Hence, we propose to improve an existing approach for image sequence interpolation by incorporating an automatic segmentation in the process, which decides in which region of the image forward or backward interpolation shall be used. Our approach is a combination of the optimal transport approach to image sequence interpolation and the segmentation by the Chan-Vese approach. We propose to solve the resulting optimality condition by a segregation loop, combined with a level set approach. We provide examples that illustrate the performance both in the interpolation error and in the human perception.

Perception-motivated interpolation of image sequences

We present a method for image interpolation that is able to create high-quality, perceptually convincing transitions between recorded images. By implementing concepts derived from human vision, the problem of a physically correct image interpolation is relaxed to that of image interpolation which is perceived as visually correct by human observers. We find that it suffices to focus on exact edge correspondences, homogeneous regions and coherent motion to compute convincing results. A user study confirms the visual quality of the proposed image interpolation approach. We show how each aspect of our approach increases perceived quality of the result. We compare the results to other methods and assess achievable quality for different types of scenes.

Structure-preserving Interpolation of Temporal and Spatial Image Sequences Using an Optical Flow-based Method

Modern tomographic imaging devices enable the acquisition of spatial and temporal image sequences. But, the spatial and temporal resolution of such devices is limited and therefore image interpolation techniques are needed to represent images at a desired level of discretization. This paper presents a method for structure-preserving interpolation between neighboring slices in temporal or spatial image sequences. In a first step, the spatiotemporal velocity field between image slices is determined using an optical flow-based registration method in order to establish spatial correspondence between adjacent slices. An iterative algorithm is applied using the spatial and temporal image derivatives and a spatiotemporal smoothing step. Afterwards, the calculated velocity field is used to generate an interpolated image at the desired time by averaging intensities between corresponding points. Three quantitative measures are defined to evaluate the performance of the interpolation method. The behavior and capability of the algorithm is demonstrated by synthetic images. A population of 17 temporal and spatial image sequences are utilized to compare the optical flow-based interpolation method to linear and shape-based interpolation. The quantitative results show that the optical flow-based method outperforms the linear and shape-based interpolation statistically significantly. The interpolation method presented is able to generate image sequences with appropriate spatial or temporal resolution needed for image comparison, analysis or visualization tasks. Quantitative and qualitative measures extracted from synthetic phantoms and medical image data show that the new method definitely has advantages over linear and shape-based interpolation.

Subpixel motion computing architecture

A pipelined optical-flow processing system that works as a virtual motion sensor has been described. It is based on a field programmable gate array (FPGA) device enabling the easy change of configuring parameters to adapt the sensor to different speeds, light conditions and other environmental factors. It is referred to as a 'virtual sensor' because it consists of a conven- tional camera as front-end supported by an FPGA processing device, which embeds the frame grabber, optical-flow algorithm implementation, output module and some configuration and storage circuitry. This is the first fully stand-alone working optical-flow processing system to include both accuracy and speed of measurement of the platform performance. The customisability of the system for different hardware resources and platforms has also been discussed, showing the resources and performance for a stand-alone board and a PCI co-processing board. Optical flow computation consists in extracting a dense velocity field from an image sequence assuming that inten- sity is conserved during displacement. This result may then be used for other applications such as 3-D reconstruction, time interpolation of image sequences, video compression, segmentation from motion, tracking, robot navigation, time-to-collision estimation and so on. The technical problem with estimating the motion of objects in 3-D is that, in the image formation process, because of the perspec- tive projection of the 3-D world onto the 2-D image plane, some of the information is lost. There are several ways of recovering the 3-D information from 2-D images using various cues. These cues are motion, binocular stereopsis, texture, shading and contour. In this paper, we will describe the implementation of a real-time motion flow system, leaving the potential applications for future studies. Optical-flow algorithms have been widely described in the literature. Some authors have addressed a comparative study of the accuracy of different approaches with synthetic sequences (1). Their evaluation using real-life sequences is difficult to address because the real optical flow of such sequences is unknown. We have focused on a classical gradient model based on Lucas and Kanade's (L & K) approach (1, 2). Several authors have emphasised the satis- factory trade-off between accuracy and efficiency in this model, which is an important factor when deciding which model is most suitable to use as a real-time processing system. For a comparative study (1), the L & K algorithm provides very accurate results, added to which, other authors specifically evaluating the efficiency against accu- racy trade-off of different optical-flow approaches (3) also regard the L&K model as being quite efficient. Finally,

A genetic algorithm approach to image sequence interpolation

Image sequence interpolation, or to obtain an up-sampled image sequence equivalently from a corresponding low-resolution image sequence, is an ill-posed inverse problem. In this study, three processing steps, namely, regularization, discretization and optimization, are used to convert the image sequence interpolation problem into a solvable optimization problem. In regularization, a fitness function combining a set of spatial and temporal performance measures for rating the quality of the interpolated (up-sampled) images is defined, which is used to convert the original ill-posed interpolation problem into a well-posed optimization problem. Discretization transforms the well-posed problem into a discrete one so that it can be solved numerically. Genetic algorithms (GAs) are used to optimize the solution in the discrete solution space using three basic operations, namely, reproduction, crossover and mutation. In the proposed approach, instead of only the spatial information within the current image frame employed in most existing methods, both the spatial and temporal information within the image sequence can be employed. Based on the experimental results obtained in this study, the interpolation results by the proposed approach are always better than those from the three existing approaches used for comparison. This shows the feasibility of the proposed approach.

Regularized iterative image interpolation and its application to spatially scalable coding

This paper presents a regularized iterative image interpolation algorithm, which can restore high frequency details in the original high resolution image. In order to apply the regularization approach to the interpolation procedure, we first present a two-dimensional separable image degradation model for a low resolution imaging system. According to the model, we propose a regularization based spatial image sequence interpolation algorithm and apply the proposed algorithm to a spatially scalable coding.

High resolution digital zoom using temporal IIR filter

We propose a set of algorithms for improving the resolution of image sequences, which are assumed to have a greater amount of information than a 2-dimensional still image. The proposed algorithms can be the theoretical basis for 3-dimensional interpolation of dynamic image sequences. More specifically, motion detection in sub-pixel level and IIR filtering along the time axis are proposed for high resolution B image magnification as well as digital image stabilization. Experimental results using real image sequence are presented which demonstrate the validity of the proposed method.

Nonlinear interpolation of TV image sequences

A 3-D nonlinear filter for the interpolation of skipped frames in the transmission of TV image sequences at low bit rates is presented. The results obtained show that the nonlinear approach is convenient in the presence of moderate movements of the objects in the observed scene